Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective

# Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective
Source: [https://arxiv.org/html/2605.21692](https://arxiv.org/html/2605.21692)
David Perera Universidade Federal de Minas Gerais Belo Horizonte, Brazil &Victor Moura Universidade Federal de Minas Gerais Belo Horizonte, Brazil &Lais Isabelle Alves dos Santos Universidade Federal de Minas Gerais Belo Horizonte, Brazil &Michel F\. C\. Haddad Queen Mary University of London London, United Kingdom &Flavio Figueiredo Universidade Federal de Minas Gerais Belo Horizonte, Brazil

###### Abstract

Characterizing precisely the asymptotic generalization error of neural networks using parameters that can be estimated efficiently is a crucial problem in machine learning, which relies heavily on heuristics and practitioners’ intuition to make key design choices\. In order to mitigate this issue, we introduce the Representation Gap, a metric closely related to the generalization error, but admitting better\-behaved asymptotic dynamics\. Focusing on equivariant diffusion models and leveraging results from optimal quantization and point\-process theory, we derive a precise asymptotic equivalent of the Representation Gap and show that it is governed by a single parameter, theintrinsic dimensionof the task, which is easy to interpret, efficient to estimate, and can be linked to the equivariances of common neural network architectures\. We show that this asymptotic dynamic also extends to a broader range of tasks and training algorithms\. Finally, we demonstrate empirically that our asymptotic law and intrinsic dimension estimation are accurate on a wide range of synthetic datasets, where these quantities are known, as well as on more realistic datasets, where we obtain results consistent with the related literature\.111Code for reproducing our experiments is available at[https://github\.com/daperera/representation\_gap](https://github.com/daperera/representation_gap)\.

## 1Introduction

Neural networks combine strong memorization capabilities with architectural and optimization biases that shape their behavior outside of the training dataset\(Hornik,[1991](https://arxiv.org/html/2605.21692#bib.bib110); Kaplanet al\.,[2020b](https://arxiv.org/html/2605.21692#bib.bib109); Kuboet al\.,[2019](https://arxiv.org/html/2605.21692#bib.bib117); Zhanget al\.,[2021](https://arxiv.org/html/2605.21692#bib.bib85)\)\. In practice, these inductive biases are often aligned with the geometry and symmetries of real\-world tasks\(Feffermanet al\.,[2016](https://arxiv.org/html/2605.21692#bib.bib134); Chianget al\.,[2022](https://arxiv.org/html/2605.21692#bib.bib189); Teneyet al\.,[2024](https://arxiv.org/html/2605.21692#bib.bib190)\)\. As a result, neural networks effectively augment the training data and can generalize well beyond simple memorization\(Zhanget al\.,[2021](https://arxiv.org/html/2605.21692#bib.bib85); Allen\-Zhuet al\.,[2019](https://arxiv.org/html/2605.21692#bib.bib126); Belkin,[2021](https://arxiv.org/html/2605.21692#bib.bib149); Simonet al\.,[2026](https://arxiv.org/html/2605.21692#bib.bib13)\)\. Recent work on diffusion models has even shown that the outputs of trained equivariant architectures can be predicted accurately from the training data and the symmetries of the model alone\(Kamb and Ganguli,[2025](https://arxiv.org/html/2605.21692#bib.bib88); Finnet al\.,[2025](https://arxiv.org/html/2605.21692#bib.bib191)\)\. These observations suggest that the generalization capabilities of neural networks are largely determined by the geometry of the data manifold and the symmetries of the model\. Our goal in this paper is to characterize neural network generalization from this geometric perspective, using measurable properties of the data and the model\.

Equivariant architectures are typically analyzed by controlling the generalization error with PAC and generalization bounds\(Chenet al\.,[2020](https://arxiv.org/html/2605.21692#bib.bib203); Elesedy and Zaidi,[2021](https://arxiv.org/html/2605.21692#bib.bib194); Tahmasebi and Jegelka,[2023](https://arxiv.org/html/2605.21692#bib.bib196)\)\. However, these bounds are not always tight, and often depend on quantities that are difficult to estimate in practice \(e\.g\. intrinsic dimension\(Ansuiniet al\.,[2019](https://arxiv.org/html/2605.21692#bib.bib200); Gonget al\.,[2019](https://arxiv.org/html/2605.21692#bib.bib198)\)\)\. Moreover, the generalization error is well defined for prediction tasks, but harder to extend to other tasks such as generative modeling\(Theiset al\.,[2015](https://arxiv.org/html/2605.21692#bib.bib207)\)\.

Motivated by these limitations, we introduce therepresentation gapℛ​\(Ω,Ωf\)\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\), which measures the discrepancy between the data manifoldΩ\\Omegaand the prediction spaceΩf\\Omega\_\{f\}of a trained modelff\. The representation gap extends the generalization error to prediction tasks and generative modeling within a unified framework\. We demonstrate that it admits a surprisingly simple asymptotic scaling inn−2/dn^\{\-2/d\}, wherennis the size of the training dataset𝔻\\mathbb\{D\}andddis anintrinsic dimensionparameter that depends only on the geometry ofΩ\\Omegaand the symmetries offf\. As a corollary, we demonstrate how model equivariance reduces this intrinsic dimensiondd, thereby provably improving generalization\.

Theoretical analyses of generalization typically assume that training and test data arei\.i\.d\.\(Shalev\-Shwartz and Ben\-David,[2014](https://arxiv.org/html/2605.21692#bib.bib204)\), and we follow this standard framework\. However, since real\-world datasets are often collected with the goal of covering the diversity of the task\(Denget al\.,[2009](https://arxiv.org/html/2605.21692#bib.bib144); Linet al\.,[2014a](https://arxiv.org/html/2605.21692#bib.bib209); Torralba and Efros,[2011](https://arxiv.org/html/2605.21692#bib.bib208)\), we also formulate our results for optimally diverse datasets\(Zador,[1982](https://arxiv.org/html/2605.21692#bib.bib179)\)\. Interestingly, we show thati\.i\.d\.datasets exhibit the same asymptotic behavior as optimally diverse datasets, up to a rescaling of the effective sample sizeneffn\_\{\\mathrm\{eff\}\}\. Overall, we make the following contributions\.

We introduce the representation gap, a geometric quantity that extends the generalization error to prediction tasks and generative modeling within a unified framework\.

We derive precise asymptotic equivalents of the representation gapfor equivariant diffusion models\. We extend this result to the setting of supervised prediction, and establish bounds relating representation gap and generalization error\. Our results hold both fori\.i\.d\.datasets and optimally diverse datasets\.

We show that asymptotic representation gap is governed by the intrinsic dimension of the task, a single parameter determined by the geometry of the data manifold and the symmetries of the model\. We further show that this intrinsic dimension can be estimated efficiently\.

We validate our theoretical predictionson controlled synthetic environments with known intrinsic dimension, as well as on more realistic datasets\.

## 2Related work

Geometric perspective on generalization\. Building on the manifold hypothesis\(Bengioet al\.,[2013](https://arxiv.org/html/2605.21692#bib.bib135)\), several works have studied neural networks as manifold learners\(Loaiza\-Ganemet al\.,[2024](https://arxiv.org/html/2605.21692#bib.bib152); Schuster and Krogh,[2021](https://arxiv.org/html/2605.21692#bib.bib151)\)\. Focusing on ReLU networks, the authors ofYaoet al\.\([2024](https://arxiv.org/html/2605.21692#bib.bib150)\)derive generalization bounds based on geometric properties of the data manifold, such as its dimension or Betti numbers\. In contrast, we derive precise asymptotic equivalents and relate them to model equivariances\. We further compare our intrinsic dimension estimator with prior manifold dimension estimators\(Popeet al\.,[2021](https://arxiv.org/html/2605.21692#bib.bib195); Gonget al\.,[2019](https://arxiv.org/html/2605.21692#bib.bib198); Ansuiniet al\.,[2019](https://arxiv.org/html/2605.21692#bib.bib200)\)and obtain consistent estimates across several datasets \(see Section[5\.2](https://arxiv.org/html/2605.21692#S5.SS2)\)\.

Generalization of equivariant neural networks\. Empirical studies have shown that equivariance improves generalization and sample efficiency\(Cohen and Welling,[2016](https://arxiv.org/html/2605.21692#bib.bib145); Bulusuet al\.,[2022](https://arxiv.org/html/2605.21692#bib.bib153)\)\. A large body of work studies PAC and generalization bounds\(Sannaiet al\.,[2021](https://arxiv.org/html/2605.21692#bib.bib154); Chenet al\.,[2020](https://arxiv.org/html/2605.21692#bib.bib203); Elesedy and Zaidi,[2021](https://arxiv.org/html/2605.21692#bib.bib194)\)\. Closest to our work,Tahmasebi and Jegelka \([2023](https://arxiv.org/html/2605.21692#bib.bib196)\)show that the generalization error of Kernel Ridge Regression is bounded byn−s/\(s\+d/2\)n^\{\-s/\(s\+d/2\)\}, whereddis the dimension of the quotient manifold induced by the model symmetries\. In contrast, we derive asymptotic equivalents for the representation gap and recover related bounds on the generalization error as a corollary \(see Section[4\.5](https://arxiv.org/html/2605.21692#S4.SS5)\)\. Finally,Kamb and Ganguli \([2025](https://arxiv.org/html/2605.21692#bib.bib88)\)derive analytic expressions for the predictions of trained diffusion models, which underlies part of our analysis\.

Scaling laws\. Our work is related to neural scaling laws\(Kaplanet al\.,[2020a](https://arxiv.org/html/2605.21692#bib.bib184)\)and recent studies on diffusion model scaling\(Meiet al\.,[2024](https://arxiv.org/html/2605.21692#bib.bib187); Liet al\.,[2024a](https://arxiv.org/html/2605.21692#bib.bib186); Lianget al\.,[2024](https://arxiv.org/html/2605.21692#bib.bib185)\)\. While prior work mainly studies empirical scaling with respect to compute, we focus on the geometric scaling induced by dataset size and model equivariance\.

Optimal quantization and point processes\.Our analysis relies on point process theory in thei\.i\.d\.setting\(Biau and Devroye,[2015](https://arxiv.org/html/2605.21692#bib.bib7); Penrose and Yukich,[2013](https://arxiv.org/html/2605.21692#bib.bib8)\)and optimal quantization theory in the optimally diverse setting\(Gruber,[2001](https://arxiv.org/html/2605.21692#bib.bib92)\)\. However, the representation gap combines geometric and statistical aspects, requiring substantial adaptation of existing results\.

## 3An illustrative example

![Refer to caption](https://arxiv.org/html/2605.21692v1/figures/diffusion_sphere_12pt.png)![Refer to caption](https://arxiv.org/html/2605.21692v1/figures/diffusion_sphere_12pt_invariant.png)\(a\) Non\-equivariant model\(b\) Equivariant modelFigure 1:Illustration of the virtual augmentation of a dataset by an equivariant diffusion model\. Plot \(a\) shows samples from a trained diffusion model, and plot \(b\) shows samples from a trained equivariant diffusion model \(with rotational invariance along thexx\-axis\)\. In both plots, the shapeΩ\\Omegais indicated by a dense cloud of red dots, the coarse dataset𝔻\\mathbb\{D\}by crosses, and the approximated shapeΩf\\Omega\_\{f\}by a dense cloud of blue dots sampled from the trained diffusion modelff\.![Refer to caption](https://arxiv.org/html/2605.21692v1/figures/first_plot.png)Figure 2:Log plot of the asymptotic evolution of the representation gap of a rotation\-equivariant model and a non\-equivariant model for a 2D sphere surface\. Thexx\-axis corresponds to the dataset sizenn, and theyy\-axis corresponds to the representation gap\. We observe a linear evolution, with slope−1\-1for the non\-equivariant model and−2\-2for the equivariant model\. The theoretical curves are shown using an empirical estimate of the multiplicative constant J in Eq\.[2](https://arxiv.org/html/2605.21692#S3.E2)Let us first introduce the main concepts of this paper through a concrete example\. We consider the task of generative modeling of 3D shapes\(Yanget al\.,[2019](https://arxiv.org/html/2605.21692#bib.bib132)\)\. The goal is to learn to sample pointsyyfrom a surfaceΩ⊂ℝ3\\Omega\\subset\\mathbb\{R\}^\{3\}that is described by a coarsenn\-point cloud𝔻∈Ωn\\mathbb\{D\}\\in\\Omega^\{n\}\. Diffusion models have recently achieved strong empirical performance on this task\(Liet al\.,[2024b](https://arxiv.org/html/2605.21692#bib.bib131)\)\. We denote byΩf\\Omega\_\{f\}the set of points that a trained diffusion modelffcan generate – in other words, the limit points of the denoising process\.

This setting is illustrated by Figure[1](https://arxiv.org/html/2605.21692#S3.F1)\. The surfaceΩ\\Omegais represented by a dense cloud of red dots, the coarse dataset𝔻\\mathbb\{D\}by crosses, and the prediction spaceΩf\\Omega\_\{f\}by a dense cloud of blue dots sampled from a trained diffusion modelff\. In this example, the surfaceΩ\\Omegaexhibits a rotational symmetry, which reduces the degrees of freedom of the task\. A natural way to leverage this symmetry is to use a rotation\-equivariant diffusion modelff\(Hoogeboomet al\.,[2022](https://arxiv.org/html/2605.21692#bib.bib133)\)\. Figure[1](https://arxiv.org/html/2605.21692#S3.F1)\(a\) shows the output of a non\-equivariant model, while Figure[1](https://arxiv.org/html/2605.21692#S3.F1)\(b\) shows the output of an equivariant model\.

We make the following two observations\. First, the distribution learned by the non\-equivariant neural network converges toward the empirical distribution1\|𝔻\|​∑y∈𝔻δy\\frac\{1\}\{\|\\mathbb\{D\}\|\}\\sum\_\{y\\in\\mathbb\{D\}\}\\delta\_\{y\}, so that the prediction spaceΩf\\Omega\_\{f\}coincides with the dataset𝔻\\mathbb\{D\}\. In other words,Ωf=𝔻\\Omega\_\{f\}=\\mathbb\{D\}\. In contrast, the equivariant model virtually augments the dataset𝔻\\mathbb\{D\}by the rotation groupGGunder which it is equivariant, so thatΩf=G​\(𝔻\)=\{g​\(z\)\|z∈𝔻,g∈G\}\\Omega\_\{f\}=G\(\\mathbb\{D\}\)=\\\{g\(z\)\|z\\in\\mathbb\{D\},g\\in G\\\}\.

It is clear from Figure[1](https://arxiv.org/html/2605.21692#S3.F1)that equivariance drastically improves the resolution of the prediction spaceΩf\\Omega\_\{f\}\. In order to quantify this improvement, we introduce the representation gap, a measure of how well the prediction spaceΩf\\Omega\_\{f\}approximates the data manifoldΩ\\Omegaunder a metricℓ\\elldefined on the ambient space\. In this work,ℓ\\elldenotes the squared Riemannian distance unless stated otherwise\.

###### Definition 1\(Representation gap\)\.

LetΩ\\Omegadenote the data manifold andΩf\\Omega\_\{f\}denote the model’s prediction space\. We define the representation gap as follows:

ℛ​\(Ω,Ωf\)=∫Ωinfz∈Ωfℓ​\(y,z\)​p​\(y\)​d​y\.\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)=\\int\_\{\\Omega\}\\inf\_\{z\\in\\Omega\_\{f\}\}\\ell\(y,z\)p\(y\)\\;\\mathrm\{d\}y\\;\.\(1\)

Concretely, Eq\.[1](https://arxiv.org/html/2605.21692#S3.E1)projects each sampley∈Ωy\\in\\Omegato the closest prediction pointz∈Ωfz\\in\\Omega\_\{f\}generated by the model, and averages this error across the data manifold\. It is worth noting that the representation gap is a special case of the Wasserstein distance\(Peyréet al\.,[2019](https://arxiv.org/html/2605.21692#bib.bib211)\)\(see Section[E\.2](https://arxiv.org/html/2605.21692#A5.SS2)in Appendix\), which is commonly used to compare sets, as well as a natural generalization of the quantization error, which we recover when the setΩf\\Omega\_\{f\}is discrete\(Graf and Luschgy,[2007](https://arxiv.org/html/2605.21692#bib.bib182)\)\.

Intuitively, a non\-equivariant modelffrequires information about all thedΩ=2d\_\{\\Omega\}=2dimensions of the shapeΩ\\Omegain order to approximate it from the dataset𝔻\\mathbb\{D\}\(as illustrated on the left of Figure[1](https://arxiv.org/html/2605.21692#S3.F1)\)\. On the other hand, the equivariant model only needs information along the rotational axis, with dimensiondΩ−1=1d\_\{\\Omega\}\-1=1\. More generally, for an arbitrary manifoldΩ\\Omegaand symmetry groupGG, the equivariant model only needs information about the quotient spaceΩ/G\\Omega/G, with dimensiondΩ/Gd\_\{\\Omega/G\}\. The remaining dimensions are implicitly recovered by the virtual augmentation of the dataset, sinceΩf=G​\(𝔻\)\\Omega\_\{f\}=G\(\\mathbb\{D\}\)\. This leads to the following asymptotic characterization of the representation gap, illustrated in Figure[2](https://arxiv.org/html/2605.21692#S3.F2)\.

###### Theorem 1\(Asymptotic representation gap – informal statement\)\.

See Theorems[3](https://arxiv.org/html/2605.21692#Thmtheorem3)and[4](https://arxiv.org/html/2605.21692#Thmtheorem4)\. The representation gapℛn\\mathcal\{R\}\_\{n\}of a model trained on a dataset𝔻\\mathbb\{D\}of sizennscales as

ℛn​∼n→\+∞​Jn2/d,\\mathcal\{R\}\_\{n\}\\;\\underset\{n\\rightarrow\+\\infty\}\{\\scalebox\{1\.5\}\{$\\sim$\}\}\\;\\frac\{J\}\{n^\{2/d\}\}\\;,\(2\)wheredddenotes the intrinsic dimension of the task:d=dΩd=d\_\{\\Omega\}for a non\-equivariant model ord=dΩ/Gd=d\_\{\\Omega/G\}for an equivariant model\. The constantJJadmits an analytic expression depending only on the geometry of the manifoldΩ\\Omega, the symmetry groupGGand the quotient metric onΩ/G\\Omega/G\.

The asymptotic evolution of the representation gapℛn\\mathcal\{R\}\_\{n\}is governed by the single parameterdd, which we name intrinsic dimension\. In particular, this result characterizes precisely the advantage of the equivariant model over the non\-equivariant one, since equivariance improves the asymptotic scaling wheneverdΩ/G<dΩd\_\{\\Omega/G\}<d\_\{\\Omega\}\. The next Section formalizes these observations for trained equivariant diffusion models \(Theorems[2](https://arxiv.org/html/2605.21692#Thmtheorem2)and[3](https://arxiv.org/html/2605.21692#Thmtheorem3)\) and extends our analysis to the setting of supervised prediction \(Theorem[4](https://arxiv.org/html/2605.21692#Thmtheorem4)\)\.

## 4Theoretical results

### 4\.1Preliminaries

We first consider the task of non\-conditional diffusion modeling and formalize the claims of Section[3](https://arxiv.org/html/2605.21692#S3)\. Let𝒴=ℝd𝒴\\mathcal\{Y\}=\\mathbb\{R\}^\{d\_\{\\mathcal\{Y\}\}\}denote the target space, of dimensiond𝒴d\_\{\\mathcal\{Y\}\}\. Under the manifold hypothesis\(Bengioet al\.,[2013](https://arxiv.org/html/2605.21692#bib.bib135)\), observations are assumed to lie on a low\-dimensional Riemannian manifoldΩ⊂𝒴\\Omega\\subset\\mathcal\{Y\}of dimensiondΩd\_\{\\Omega\}, whose geometry captures the symmetries of the task\. We further suppose access to a dataset𝔻⊂Ω\\mathbb\{D\}\\subset\\Omegacomposed ofnnobservations drawn from a distributionppsupported onΩ\\Omega\. We consider neural networksfθf\_\{\\theta\}in a parametric familyℱΘ⊂ℱ​\(𝒴×ℝ,𝒴\)\\mathcal\{F\}\_\{\\Theta\}\\subset\\mathcal\{F\}\(\\mathcal\{Y\}\\times\\mathbb\{R\},\\mathcal\{Y\}\), which we simply denote byffwhen there is no ambiguity\.

For simplicity, we followKamb and Ganguli \([2025](https://arxiv.org/html/2605.21692#bib.bib88)\)andFinnet al\.\([2025](https://arxiv.org/html/2605.21692#bib.bib191)\), and focus on Denoising Diffusion Implicit Models \(DDIM\) diffusion models\(Songet al\.,[2022](https://arxiv.org/html/2605.21692#bib.bib159)\)\. DDIM models are trained to reverse a stochastic forward diffusion process that incrementally adds Gaussian noise to the data distribution while shrinking data points toward the origin\. Noise addition is governed by a noise scheduleαt\\alpha\_\{t\}, witht∈\[0,T\]t\\in\[0,T\]\. At timett, the noised distribution can be writtenπt​\(y\)=1\|𝔻\|​∑z∈𝔻𝒩​\(y\|αt​z,\(1−αt\)​I\)\\pi\_\{t\}\(y\)=\\frac\{1\}\{\|\\mathbb\{D\}\|\}\\sum\_\{z\\in\\mathbb\{D\}\}\\mathcal\{N\}\(y\|\\sqrt\{\\alpha\_\{t\}\}z,\(1\-\\alpha\_\{t\}\)I\), thus interpolating between the empirical data distributionπ0=1\|𝔻\|​∑z∈𝔻δz\\pi\_\{0\}=\\frac\{1\}\{\|\\mathbb\{D\}\|\}\\sum\_\{z\\in\\mathbb\{D\}\}\\delta\_\{z\}and the isotropic Gaussian distributionπT=𝒩​\(0,I\)\\pi\_\{T\}=\\mathcal\{N\}\(0,I\)\. In this context, DDIM models are trained to approximate the score functionst=∇log⁡πts\_\{t\}=\\nabla\\log\\pi\_\{t\}using the loss

ℒ​\(θ\)=𝔼t,y0,η​‖fθ​\(αt​y0\+1−αt​η,t\)−η‖22,\\mathcal\{L\}\(\\theta\)=\\mathbb\{E\}\_\{t,y\_\{0\},\\eta\}\\\|f\_\{\\theta\}\(\\sqrt\{\\alpha\_\{t\}\}y\_\{0\}\+\\sqrt\{1\-\\alpha\_\{t\}\}\\eta,t\)\-\\eta\\\|\_\{2\}^\{2\}\\;,\(3\)wheret∼𝕌​\[0,T\]t\\sim\\mathbb\{U\}\[0,T\],y0∼π0y\_\{0\}\\sim\\pi\_\{0\}andη∼𝒩​\(0,I\)\\eta\\sim\\mathcal\{N\}\(0,I\)\. At sampling time, an initial pointyT∼𝒩​\(0,I\)y\_\{T\}\\sim\\mathcal\{N\}\(0,I\)is sampled and then updated using the deterministic flow

y˙t=−γt​\(yt\+st​\(yt\)\),\\dot\{y\}\_\{t\}=\-\\gamma\_\{t\}\(y\_\{t\}\+s\_\{t\}\(y\_\{t\}\)\)\\;,\(4\)wherettgoes backward fromTTto0\. The outputs correspond to the endpoints reachable by this reverse flow, i\.e\.:

Ωf≜\{y0\|yT∼𝒩​\(0,I\),yt​solves Eq\.[4](https://arxiv.org/html/2605.21692#S4.E4)\}\.\\Omega\_\{f\}\\triangleq\\\{y\_\{0\}\\;\|\\;y\_\{T\}\\sim\\mathcal\{N\}\(0,I\)\\;,\\;y\_\{t\}\\text\{ solves Eq\. \}\\ref\{eq:diffusion\_sampling\}\\\}\\;\.\(5\)

### 4\.2Virtual augmentation of a dataset by an equivariant model

A diffusion model minimizing the training objectiveℒ\\mathcal\{L\}exactly —\- and therefore recovering the true score functionsts\_\{t\}\-— generates samples following the empirical distributionπ0=1\|𝔻\|​∑z∈𝔻δz\\pi\_\{0\}=\\frac\{1\}\{\|\\mathbb\{D\}\|\}\\sum\_\{z\\in\\mathbb\{D\}\}\\delta\_\{z\}\(Song and Ermon,[2019](https://arxiv.org/html/2605.21692#bib.bib136)\)\. In this case, the prediction spaceΩf\\Omega\_\{f\}learned by the modelffis the training data itself\. Therefore,Ωf\\Omega\_\{f\}provides a discrete approximation of the data manifoldΩ\\Omegagiven byΩf=𝔻\\Omega\_\{f\}=\\mathbb\{D\}\.

In practice, however, the neural network familyℱΘ\\mathcal\{F\}\_\{\\Theta\}has a limited expressivity, which prevents the perfect estimation of the true scorests\_\{t\}\. Instead, neural network architectures are often designed to enforce the symmetries of the task\. Remarkably, it is possible to show followingKamb and Ganguli \([2025](https://arxiv.org/html/2605.21692#bib.bib88)\)that these architectural constraints induce a virtual augmentation of the training dataset𝔻\\mathbb\{D\}by the symmetry groupGGinduced by the architecture, so that we have in effectΩf=G​\(𝔻\)\\Omega\_\{f\}=G\(\\mathbb\{D\}\)\.

###### Theorem 2\(Virtual augmentation of a dataset by an equivariant model\)\.

See Proposition[4](https://arxiv.org/html/2605.21692#Thmprop4)in Appendix\. Letffdenote a diffusion model equivariant under a symmetry groupGGand minimizing the training objective in Eq\.[3](https://arxiv.org/html/2605.21692#S4.E3)on a dataset𝔻\\mathbb\{D\}\. Then under mild assumptions onGG,Ω\\Omegaand𝔻\\mathbb\{D\}, the set of points that can be predicted byffisΩf=G​\(𝔻\)\\Omega\_\{f\}=G\(\\mathbb\{D\}\)\.

###### Proof\.

The proof of Theorem[2](https://arxiv.org/html/2605.21692#Thmtheorem2)relies on the following observation: the score functionsts\_\{t\}at a pointy∈𝒴y\\in\\mathcal\{Y\}can be written as an integral over the orbitsG​\(𝔻\)G\(\\mathbb\{D\}\)of the dataset𝔻\\mathbb\{D\}:

st​\(y\)=−11−αt​∫G​\(𝔻\)\(y−αt​z\)​Wt​\(z\)​dz,s\_\{t\}\(y\)=\-\\frac\{1\}\{1\-\\alpha\_\{t\}\}\\int\_\{G\(\\mathbb\{D\}\)\}\(y\-\\sqrt\{\\alpha\_\{t\}\}z\)W\_\{t\}\(z\)\\mathrm\{d\}z\\;,where each pointz∈G​\(𝔻\)z\\in G\(\\mathbb\{D\}\)is weighted by the distribution

Wt​\(z\)=𝒩​\(y\|αt​z,\(1−αt\)​I\)∫G​\(𝔻\)𝒩​\(y\|αt​z′,\(1−αt\)​I\)​dz′\.W\_\{t\}\(z\)=\\frac\{\\mathcal\{N\}\\left\(y\|\\sqrt\{\\alpha\_\{t\}\}z,\(1\-\\alpha\_\{t\}\)I\\right\)\}\{\\int\_\{G\(\\mathbb\{D\}\)\}\\mathcal\{N\}\\left\(y\|\\sqrt\{\\alpha\_\{t\}\}z^\{\\prime\},\(1\-\\alpha\_\{t\}\)I\\right\)\\mathrm\{d\}z^\{\\prime\}\}\\;\.We can see thatWt​\(y\)W\_\{t\}\(y\)acts as a softmax that peaks at the minimizery∗=argminz∈G​\(𝔻\)​ℓ​\(y,z\)y^\{\*\}=\\mathrm\{argmin\}\_\{z\\in G\(\\mathbb\{D\}\)\}\\ell\(y,z\)for smalltt\. More precisely, we can use a Laplace approximation to show thatWt​\(y\)W\_\{t\}\(y\)concentrates the probability mass aroundy∗y^\{\*\}whent→0t\\rightarrow 0\.

Under the hypothesis thatffminimizes the training objective in Eq\.[3](https://arxiv.org/html/2605.21692#S4.E3), we can therefore write

f​\(yt,t\)=−11−αt​∫G​\(𝔻\)\(yt−αt​z\)​Wt​\(z\)​dz=11−αt​\(yt−yt∗\)\+o​\(11−αt\),f\(y\_\{t\},t\)=\-\\frac\{1\}\{1\-\\alpha\_\{t\}\}\\int\_\{G\(\\mathbb\{D\}\)\}\(y\_\{t\}\-\\sqrt\{\\alpha\_\{t\}\}z\)W\_\{t\}\(z\)\\mathrm\{d\}z=\\frac\{1\}\{1\-\\alpha\_\{t\}\}\(y\_\{t\}\-y\_\{t\}^\{\*\}\)\+o\\left\(\\frac\{1\}\{1\-\\alpha\_\{t\}\}\\right\)\\;,which in turn impliesyt−yt∗≈\(1−αt\)​f​\(yt,t\)→0y\_\{t\}\-y^\{\*\}\_\{t\}\\approx\(1\-\\alpha\_\{t\}\)f\(y\_\{t\},t\)\\rightarrow 0, and thereforelimt→0yt=limt→0yt∗∈G​\(𝔻\)\\lim\_\{t\\rightarrow 0\}y\_\{t\}=\\lim\_\{t\\rightarrow 0\}y\_\{t\}^\{\*\}\\in G\(\\mathbb\{D\}\)\(by properties ofGG\)\. This provesΩf⊂G​\(𝔻\)\\Omega\_\{f\}\\subset G\(\\mathbb\{D\}\)\. The reverse inclusion is detailed in Appendix\. ∎

### 4\.3Representation gap for non\-conditional diffusion

Using Theorem[2](https://arxiv.org/html/2605.21692#Thmtheorem2), we can now characterize the asymptotic representation gap in the large sample regime\. Crucially, since equivariant architectures virtually augment the dataset byΩf=G​\(𝔻\)\\Omega\_\{f\}=G\(\\mathbb\{D\}\), the representation gap no longer depends on the ambient manifoldΩ\\Omega, but only on the geometry of the quotient manifoldΩ/G\\Omega/G\. The representation gap therefore reduces to a quantization problem on the quotient manifold, allowing the use of asymptotic results from optimal quantization and point processes theory\.

The representation gapℛ​\(Ω,Ωf\)\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)depends on how the dataset𝔻\\mathbb\{D\}is sampled fromΩ\\Omega\. It is typical to assume that𝔻\\mathbb\{D\}is a dataset of sizennsampledi\.i\.d\.from the data distributionpp, and we denote

ℛn≜ℛ​\(Ω,Ωf​\(𝔻\)\)\\mathcal\{R\}\_\{n\}\\triangleq\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\(\\mathbb\{D\}\)\)\(6\)the correspondingrandom representation gap\. In practice, however, datasets are often collected to cover the diversity of the task, modulo its known invariants\(Torralba and Efros,[2011](https://arxiv.org/html/2605.21692#bib.bib208)\)\. Motivated by this observation, we also consider the setting where𝔻\\mathbb\{D\}is optimally diverse, i\.e\. minimizes the representation gap, and denote theoptimal representation gapby

ℛn∗≜inf𝔻⊂Ω,\|𝔻\|=nℛ​\(Ω,Ωf​\(𝔻\)\)\\mathcal\{R\}\_\{n\}^\{\*\}\\triangleq\\inf\_\{\\mathbb\{D\}\\subset\\Omega,\\;\|\\mathbb\{D\}\|=n\}\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\(\\mathbb\{D\}\)\)\(7\)
The following result is a formalization of Theorem[1](https://arxiv.org/html/2605.21692#Thmtheorem1)\.

###### Theorem 3\(Representation gap for non\-conditional diffusion\)\.

See Propositions[1](https://arxiv.org/html/2605.21692#Thmprop1),[2](https://arxiv.org/html/2605.21692#Thmprop2),[3](https://arxiv.org/html/2605.21692#Thmprop3)and[5](https://arxiv.org/html/2605.21692#Thmprop5)in Appendix\. Letffdenote an equivariant model satisfyingΩf=G​\(𝔻\)\\Omega\_\{f\}=G\(\\mathbb\{D\}\), where𝔻\\mathbb\{D\}is a dataset of sizenn\. Suppose further that the orbitsG​\(y\)G\(y\)have constant volume for each pointy∈Ωy\\in\\Omega\. Then under mild regularity assumptions onΩ\\OmegaandGG, the representation gap satisfies

\(i\.i\.d\.\)ℛn\\displaystyle\\text\{\(i\.i\.d\.\)\}\\quad\\mathcal\{R\}\_\{n\}∼ℙJdn2/d\(optimal\)ℛn⋆∼Jd∗n2/d\\displaystyle\\sim\_\{\\mathbb\{P\}\}\\frac\{J\_\{d\}\}\{n^\{2/d\}\}\\qquad\\qquad\\text\{\(optimal\)\}\\quad\\mathcal\{R\}\_\{n\}^\{\\star\}\\sim\\frac\{J\_\{d\}^\{\*\}\}\{n^\{2/d\}\}\(8\)whered=dΩ/Gd=d\_\{\\Omega/G\}denotes the dimension ofΩ/G\\Omega/G, the quotient space ofΩ\\Omegaby the symmetry groupGG, and the constantsJdJ\_\{d\}andJd∗J\_\{d\}^\{\*\}depend only on the quotient geometry and data distribution onΩ/G\\Omega/G\. In particular, equivariance improves the asymptotic scaling wheneverdΩ/G<dΩd\_\{\\Omega/G\}<d\_\{\\Omega\}\.

###### Proof\.

We haveΩf=G​\(𝔻\)\\Omega\_\{f\}=G\(\\mathbb\{D\}\)\. Using the orbit decomposition ofΩ\\Omegaand the isometric action ofGG\(see for instanceGallotet al\.\([1990](https://arxiv.org/html/2605.21692#bib.bib96)\)\), the representation gap reduces to

ℛ​\(Ω,Ωf\)=\|G\|​∫Ω/Gminz∈𝔻⁡ℓΩ/G​\(y,z\)​p​\(y\)​dy,\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)=\|G\|\\int\_\{\\Omega/G\}\\min\_\{z\\in\\mathbb\{D\}\}\\ell\_\{\\Omega/G\}\(y,z\)\\,p\(y\)\\,\\mathrm\{d\}y,so that the problem reduces to quantization on the quotient manifoldΩ/G\\Omega/G\. The asymptotic optimal representation gap then follows from Zador’s theorem \(see Theorem 2 inGruber \([2001](https://arxiv.org/html/2605.21692#bib.bib92)\)\)\. Thei\.i\.d\.setting is treated by Proposition[3](https://arxiv.org/html/2605.21692#Thmprop3)in Appendix\. ∎

Theorem[3](https://arxiv.org/html/2605.21692#Thmtheorem3)provides a precise asymptotic equivalent of the representation gap, which is remarkable since most existing analyses of neural network generalization focus on bounds\(Zhanget al\.,[2021](https://arxiv.org/html/2605.21692#bib.bib85)\)\. In particular, the convergence in probability on the left of Eq\.[8](https://arxiv.org/html/2605.21692#S4.E8)is a strong result, which implies that the representation gap of ani\.i\.d\.dataset is asymptotic close to its equivalentJd​n−2/dJ\_\{d\}\\,n^\{\-2/d\}with arbitrarily high probability\. The constantsJdJ\_\{d\}andJd⋆J\_\{d\}^\{\\star\}admit analytic expressions \(see Prop\.[2](https://arxiv.org/html/2605.21692#Thmprop2)and[3](https://arxiv.org/html/2605.21692#Thmprop3)in Appendix\)\.

As a direct corollary, we obtainℛn∼ℙℛneff⋆\\mathcal\{R\}\_\{n\}\\sim\_\{\\mathbb\{P\}\}\\mathcal\{R\}\_\{n\_\{\\mathrm\{eff\}\}\}^\{\\star\}, with effective sample sizeneff=\(Jd⋆/Jd\)d/2n\_\{\\mathrm\{eff\}\}=\\left\(J\_\{d\}^\{\\star\}/J\_\{d\}\\right\)^\{d/2\}\. Thus, random datasets exhibit the same asymptotic behavior as optimally diverse datasets, up to a rescaling of the effective sample sizeneffn\_\{\\mathrm\{eff\}\}\.

As a corollary of Theorem[2](https://arxiv.org/html/2605.21692#Thmtheorem2), Theorem[3](https://arxiv.org/html/2605.21692#Thmtheorem3)applies to equivariant diffusion models minimizing the training objective in Eq\.[3](https://arxiv.org/html/2605.21692#S4.E3), under the regularity assumptions of Theorem[2](https://arxiv.org/html/2605.21692#Thmtheorem2)\. However, the result applies more generally to any generative model satisfyingΩf=G​\(𝔻\)\\Omega\_\{f\}=G\(\\mathbb\{D\}\), and is therefore not restricted to diffusion models\. It also extends naturally to conditional generative models when the conditioning variable takes finitely many values \(see Proposition[6](https://arxiv.org/html/2605.21692#Thmprop6)in Appendix\)\.

### 4\.4Representation gap for supervised prediction

We now turn to the more general setting of supervised prediction\. Each inputx∈Ω𝒳x\\in\\Omega\_\{\\mathcal\{X\}\}is associated with a unique targety​\(x\)∈Ω𝒴y\(x\)\\in\\Omega\_\{\\mathcal\{Y\}\}, so that the observation manifoldΩ⊂Ω𝒳×Ω𝒴\\Omega\\subset\\Omega\_\{\\mathcal\{X\}\}\\times\\Omega\_\{\\mathcal\{Y\}\}can be identified with the graph of the functiony:Ω𝒳→Ω𝒴y:\\Omega\_\{\\mathcal\{X\}\}\\rightarrow\\Omega\_\{\\mathcal\{Y\}\}\. In particular, the intrinsic dimension ofΩ\\Omegacoincides with that of the input manifold,i\.e\.,dΩ=dΩ𝒳d\_\{\\Omega\}=d\_\{\\Omega\_\{\\mathcal\{X\}\}\}, independently of the dimension of𝒴\\mathcal\{Y\}\. We further assume that the modelffgenerates a unique predictionf​\(x\)f\(x\)for each inputx∈Ω𝒳x\\in\\Omega\_\{\\mathcal\{X\}\}, so that the prediction manifoldΩf\\Omega\_\{f\}can similarly be identified with the graph offf\. In this context, the conditional representation gap is defined by

ℛ​\(Ω,Ωf\)=∫Ω𝒳minz′∈Ωf⁡ℓ​\(z,z′\)​p​\(x\)​𝑑x,\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)=\\int\_\{\\Omega\_\{\\mathcal\{X\}\}\}\\min\_\{z^\{\\prime\}\\in\\Omega\_\{f\}\}\\ell\(z,z^\{\\prime\}\)\\,p\(x\)\\,dx,\(9\)wherez=\(x,y​\(x\)\)∈Ωz=\(x,y\(x\)\)\\in\\Omegaandz′=\(x′,f​\(x′\)\)∈Ωfz^\{\\prime\}=\(x^\{\\prime\},f\(x^\{\\prime\}\)\)\\in\\Omega\_\{f\}\. We further denote byℓ𝒳\\ell\_\{\\mathcal\{X\}\}andℓ𝒴\\ell\_\{\\mathcal\{Y\}\}the metrics induced byℓ\\ellon𝒳\\mathcal\{X\}and𝒴\\mathcal\{Y\}respectively\.

###### Theorem 4\(Conditional representation gap of an equivariant model\)\.

See Proposition[7](https://arxiv.org/html/2605.21692#Thmprop7)\. Letffdenote an equivariantLL\-Lipschitz model satisfyingΩf=G​\(𝔻\)\\Omega\_\{f\}=G\(\\mathbb\{D\}\), where𝔻\\mathbb\{D\}is a training dataset of sizennandL\>0L\>0\. Suppose further that the orbitsG​\(x\)G\(x\)have constant volume for each pointx∈Ω𝒳x\\in\\Omega\_\{\\mathcal\{X\}\}, and that the metricℓ\\ellis additively separable on𝒳\\mathcal\{X\}and𝒴\\mathcal\{Y\}\. Then under mild regularity assumptions, the representation gap satisfies

\(i\.i\.d\.\)ℛn=Oℙ​\(1n2/d\)\(optimal\)ℛn⋆=O​\(1n2/d\),\\begin\{aligned\} \\text\{\(i\.i\.d\.\)\}\\quad\\mathcal\{R\}\_\{n\}&=O\_\{\\mathbb\{P\}\}\\left\(\\frac\{1\}\{n^\{2/d\}\}\\right\)\\qquad\\qquad\\text\{\(optimal\)\}\\quad\\mathcal\{R\}\_\{n\}^\{\\star\}&=O\\left\(\\frac\{1\}\{n^\{2/d\}\}\\right\)\\end\{aligned\}\\;,\(10\)whereΩ𝒳/G\\Omega\_\{\\mathcal\{X\}\}/Gdenotes the quotient space ofΩ𝒳\\Omega\_\{\\mathcal\{X\}\}by the symmetry groupGG, andd=dΩ𝒳/Gd=d\_\{\\Omega\_\{\\mathcal\{X\}\}/G\}denotes the dimension ofΩ𝒳/G\\Omega\_\{\\mathcal\{X\}\}/G\.

###### Proof\.

Using the equivariance offf, we proceed as in the proof of Theorem[3](https://arxiv.org/html/2605.21692#Thmtheorem3)and reduce the representation gap to a quantization problem on the quotient manifoldΩ𝒳/G\\Omega\_\{\\mathcal\{X\}\}/G\. Then, letz=\(x,y​\(x\)\)∈Ωz=\(x,y\(x\)\)\\in\\Omegadenote a data sample with inputxx, letx^=argminx′∈𝔻𝒳​ℓ𝒳​\(x,x′\)\\hat\{x\}=\\mathrm\{argmin\}\_\{x^\{\\prime\}\\in\\mathbb\{D\}\_\{\\mathcal\{X\}\}\}\\ell\_\{\\mathcal\{X\}\}\(x,x^\{\\prime\}\)denote the nearest training input toxx, and letz^=\(x^,y​\(x^\)\)\\hat\{z\}=\(\\hat\{x\},y\(\\hat\{x\}\)\)denote the corresponding training sample\. Sinceffinterpolates the training dataset, we havef​\(x^\)=y​\(x^\)f\(\\hat\{x\}\)=y\(\\hat\{x\}\)\. Using the additive separability ofℓ\\elland the Lipschitzness offf,

ℓ​\(z,z^\)≤ℓ𝒳​\(x,x^\)\+ℓ𝒴​\(f​\(x\),f​\(x^\)\)≤\(1\+L\)​ℓ𝒳​\(x,x^\)\.\\ell\\big\(z,\\hat\{z\}\\big\)\\leq\\ell\_\{\\mathcal\{X\}\}\(x,\\hat\{x\}\)\+\\ell\_\{\\mathcal\{Y\}\}\(f\(x\),f\(\\hat\{x\}\)\)\\leq\(1\+L\)\\ell\_\{\\mathcal\{X\}\}\(x,\\hat\{x\}\)\\;\.
Thus,ℛ​\(Ω,Ωf\)≤\(1\+L\)​∫Ω𝒳minx′∈𝔻𝒳⁡ℓ𝒳​\(x,x′\)​p​\(x\)​𝑑x,\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)\\leq\(1\+L\)\\int\_\{\\Omega\_\{\\mathcal\{X\}\}\}\\min\_\{x^\{\\prime\}\\in\\mathbb\{D\}\_\{\\mathcal\{X\}\}\}\\ell\_\{\\mathcal\{X\}\}\(x,x^\{\\prime\}\)\\,p\(x\)\\,dx,and the result follows from Theorem[3](https://arxiv.org/html/2605.21692#Thmtheorem3)\. ∎

### 4\.5Comparison with generalization error

A natural question is to relate the representation gapR​\(Ω,Ωf\)R\(\\Omega,\\Omega\_\{f\}\)to the generalization error\(Shalev\-Shwartz and Ben\-David,[2014](https://arxiv.org/html/2605.21692#bib.bib204)\), commonly used to characterize generalization\. We focus on the setting of prediction tasks, for which there is a widely accepted definition of the generalization error,ℰ=∫Ωℓ𝒴​\(y​\(x\),f​\(x\)\)​p​\(x\)​dx\\mathcal\{E\}=\\int\_\{\\Omega\}\\ell\_\{\\mathcal\{Y\}\}\(y\(x\),f\(x\)\)\\,p\(x\)\\,\\mathrm\{d\}x\.

###### Theorem 5\(Comparison with generalization error\)\.

See Proposition[8](https://arxiv.org/html/2605.21692#Thmprop8)in Appendix\. If the modelffisLL\-Lipschitz and the metricℓ\\ellis additively separable on𝒳\\mathcal\{X\}and𝒴\\mathcal\{Y\}, we have under mild regularity assumptions that

11\+L​ℰ≤ℛ​\(Ω,Ωf\)≤ℰ\.\\frac\{1\}\{1\+L\}\\mathcal\{E\}\\leq\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)\\leq\\mathcal\{E\}\\;\.\(11\)

Combining Theorems[4](https://arxiv.org/html/2605.21692#Thmtheorem4)and[5](https://arxiv.org/html/2605.21692#Thmtheorem5), we obtainℰ=O​\(n−2/dΩ\)\\mathcal\{E\}=O\\left\(n^\{\-2/d\_\{\\Omega\}\}\\right\)asn→\+∞n\\rightarrow\+\\infty, a result closely related toTahmasebi and Jegelka \([2023](https://arxiv.org/html/2605.21692#bib.bib196)\)\. Moreover,ℛ​\(Ω,Ωf\)=0\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)=0impliesf​\(x\)=yxf\(x\)=y\_\{x\}almost everywhere, and thereforeℰ=0\\mathcal\{E\}=0\. Generalization error and representation gap are therefore closely related\.

## 5Experimental results

We now validate experimentally the theoretical results of Section[4](https://arxiv.org/html/2605.21692#S4)\.

HypercubeWave![Refer to caption](https://arxiv.org/html/2605.21692v1/figures/diffusion_cube_4pt.png)![Refer to caption](https://arxiv.org/html/2605.21692v1/figures/diffusion_cube_4pt_invariant.png)![Refer to caption](https://arxiv.org/html/2605.21692v1/figures/diffusion_wave_8pt.png)![Refer to caption](https://arxiv.org/html/2605.21692v1/figures/diffusion_wave_8pt_invariant.png)\(a\) Normal\(b\) Equivariant\(c\) Normal\(d\) EquivariantFigure 3:Virtual augmentation of a dataset in the non\-conditional setting \(Hypercube dataset\) and the conditional setting \(Wave dataset\)\. Plots \(a\) and \(c\) show samples from a trained non\-equivariant diffusion model, while plots \(b\) and \(d\) show samples from a trained equivariant diffusion model \(with translation invariance along theyy\-axis\)\. For the Wave dataset, the conditioning corresponds to thexx\-axis\.Datasets\.We conduct experiments on several synthetic datasets for non\-conditional generative modeling and supervised prediction\. They are illustrated in Figure[1](https://arxiv.org/html/2605.21692#S3.F1)and Figure[3](https://arxiv.org/html/2605.21692#S5.F3)\.

Hypercubecorresponds to adΩd\_\{\\Omega\}\-dimensional hypercubeΩ=\[−c2,c2\]dΩ\\Omega=\\left\[\-\\frac\{c\}\{2\},\\frac\{c\}\{2\}\\right\]^\{d\_\{\\Omega\}\}of sideccembedded into ad𝒴d\_\{\\mathcal\{Y\}\}ambient space\. This dataset is translation\-invariant along each dimension\.

Hyperspherecorresponds to a 2\-dimensional hypersphere∂B​\(0,r\)\\partial B\(0,r\)of radiusrrembedded into a 3\-dimensional ambient space\. This dataset features rotational invariances along axesxx,yyandzz\.

Waveis a 2\-dimensional wave surface embedded into a 3\-dimensional ambient space, obtained by translating a\(x,z\)\(x,z\)\-curve composed of half\-circles along theyy\-axis\. This dataset corresponds to a conditional prediction task, wherexxis the input and\(y,z\)\(y,z\)is the target\. It features translation invariance overyy\.

![Refer to caption](https://arxiv.org/html/2605.21692v1/figures/diffusion_representation_gap_cube_2d.png)\(a\) Hypercube![Refer to caption](https://arxiv.org/html/2605.21692v1/figures/diffusion_representation_gap_wave_2d.png)\(b\) WaveFigure 4:Asymptotic behavior of the representation gap across the two datasets of Figure[3](https://arxiv.org/html/2605.21692#S5.F3)\. Thexx\-axis corresponds to the number of training pointsnn, and theyy\-axis corresponds to the representation gap\. We plot the theoretical loss in Eq\.[8](https://arxiv.org/html/2605.21692#S4.E8)\(green\), the optimal representation gapℛn∗\\mathcal\{R\}^\{\*\}\_\{n\}\(blue\) and the empirical representation gapℛn​\(Ω,Ωf\)\\mathcal\{R\}\_\{n\}\(\\Omega,\\Omega\_\{f\}\)computed from a diffusion modelfftrained on an optimal dataset𝔻\\mathbb\{D\}\(orange\)\.Swiss rollandDeformed sphereare popular 2\-dimensional manifolds embedded into a 3\-dimensional ambient spaces\(Jacobsenet al\.,[2025](https://arxiv.org/html/2605.21692#bib.bib210)\)\.

Real Data:FollowingPopeet al\.\([2021](https://arxiv.org/html/2605.21692#bib.bib195)\), we also consider the datasetsMNIST\(Lecunet al\.,[1998](https://arxiv.org/html/2605.21692#bib.bib174)\), SVHN\(Netzeret al\.,[2011](https://arxiv.org/html/2605.21692#bib.bib3)\), CIFAR10\(Krizhevsky and Hinton,[2009](https://arxiv.org/html/2605.21692#bib.bib4)\), MSCOCO\(Linet al\.,[2014b](https://arxiv.org/html/2605.21692#bib.bib5)\), and Tiny\-ImageNet\(Leet al\.,[2015](https://arxiv.org/html/2605.21692#bib.bib6)\)\. which are standard benchmarks for machine learning and intrinsic dimension estimation\(Popeet al\.,[2021](https://arxiv.org/html/2605.21692#bib.bib195); Ansuiniet al\.,[2019](https://arxiv.org/html/2605.21692#bib.bib200)\)\.

Architecture\.For the non\-conditional task, we use a three\-layer MLP\(Rumelhartet al\.,[1986](https://arxiv.org/html/2605.21692#bib.bib139)\)with ReLU activation and 128 hidden units\. For the conditional task, we use a 10\-layer MLP with SiLU activation\(Ramachandranet al\.,[2017](https://arxiv.org/html/2605.21692#bib.bib140)\), 128 hidden units, residual connections, and linear embedding for the conditioning\. Model equivariance is enforced on top of this architecture by input normalization and prediction shift\.

Training and optimization\.For the synthetic experiments, we use a DDIM diffusion model\(Songet al\.,[2022](https://arxiv.org/html/2605.21692#bib.bib159)\), trained with a linear temperature schedule withT=100T=100steps\. We use theℒ2\\mathcal\{L\}\_\{2\}loss defined on the ambient space𝒴\\mathcal\{Y\}\. The models are trained with the Adam optimizer\(Kingma and Ba,[2017](https://arxiv.org/html/2605.21692#bib.bib142)\)for 50000 steps, with learning rateλ=10−3\\lambda=10^\{\-3\}\.

Metric\.To estimate the representation gap, we sample 1000 points from the trained model and 1000 points uniformly from a hold\-out set onΩ\\Omega\. We then approximate Eq\.[1](https://arxiv.org/html/2605.21692#S3.E1)by computing the average minimum distance between these two clouds of points\.

Training point selection\.We experiment with bothi\.i\.d\.sampling and optimally diverse sampling \(see Section[4](https://arxiv.org/html/2605.21692#S4)\)\. For simple synthetic datasets such as Hypercube and HyperSphere, the optimal samples admit an analytic expression that we can use\. For the remaining synthetic datasets, we estimate the optimal samples empirically by minimizing Eq\.[1](https://arxiv.org/html/2605.21692#S3.E1)\. Although this optimization problem is NP\-hard\(Aloiseet al\.,[2009](https://arxiv.org/html/2605.21692#bib.bib2)\), standard optimization algorithms such as K\-means\+\+\(Arthur and Vassilvitskii,[2006](https://arxiv.org/html/2605.21692#bib.bib212)\)and discrete Lloyd on\-manifold centroid snapping\(Lloyd,[1982](https://arxiv.org/html/2605.21692#bib.bib213)\)were sufficient in practice\.

Intrinsic dimension estimation\.In order to estimate the intrinsic dimension of a task, we evaluate the random representation gapℛn\\mathcal\{R\}\_\{n\}for several values ofnn, fit a linear model to the resulting points\(log⁡\(n\),log⁡\(ℛn\)\)\(\\log\(n\),\\log\(\\mathcal\{R\}\_\{n\}\)\), and extract the slope, which equals−2/d\-2/daccording to Theorem[3](https://arxiv.org/html/2605.21692#Thmtheorem3)\. We can alternatively use the optimal representation gapℛn∗\\mathcal\{R\}\_\{n\}^\{\*\}, in which case the optimal samples are selected by empirically minimizing Eq\.[1](https://arxiv.org/html/2605.21692#S3.E1)as described above\. We observed that this second estimator converged faster in practice, often with as few asn=50n=50samples\. As a corollary of Theorem[1](https://arxiv.org/html/2605.21692#Thmtheorem1), Eq\.[2](https://arxiv.org/html/2605.21692#S3.E2)naturally defines an estimator of the intrinsic dimension of the data manifoldΩ\\Omega\. To this end, we estimate the representation gap directly from the training points𝔻\\mathbb\{D\}instead of using the model predictionsΩf\\Omega\_\{f\}\. We repeat intrinsic dimension estimation over 5 random seeds and report the mean and standard deviation\.

### 5\.1Validating Theorems 2 and 3

Figures[1](https://arxiv.org/html/2605.21692#S3.F1)and[3](https://arxiv.org/html/2605.21692#S5.F3)highlight two distinct regimes\. Non\-equivariant models converge toward the training datasetΩf=𝔻\\Omega\_\{f\}=\\mathbb\{D\}, while equivariant models converge toward the virtually augmented datasetΩf=G​\(𝔻\)\\Omega\_\{f\}=G\(\\mathbb\{D\}\)\. This behavior is consistently observed across different geometries and experimentally validates the claim of Theorem[2](https://arxiv.org/html/2605.21692#Thmtheorem2)\.

To further validate the asymptotic formula of Theorem[3](https://arxiv.org/html/2605.21692#Thmtheorem3), we evaluate the representation gap on several manifoldsΩ\\Omega\. We report the results in Figures[2](https://arxiv.org/html/2605.21692#S3.F2)and[4](https://arxiv.org/html/2605.21692#S5.F4)\. Across all datasets, the empirical representation gap closely follows the predicted asymptotic scaling\. Moreover, convergence toward the asymptotic regime occurs rapidly, in some cases with as few asn=50n=50samples\. This suggests that the asymptotic analysis remains relevant in practical settings, including applications where only limited amounts of training data are available\.

### 5\.2Intrinsic dimension estimation

Table 1:Estimated intrinsic dimension on synthetic and real\-world datasets\. On synthetic data, bothℛn\\mathcal\{R\}\_\{n\}andℛn⋆\\mathcal\{R\}\_\{n\}^\{\\star\}recover the true dimension accurately\. On real\-world datasets, our estimates are consistent with estimates from prior work\(Popeet al\.,[2021](https://arxiv.org/html/2605.21692#bib.bib195); Ansuiniet al\.,[2019](https://arxiv.org/html/2605.21692#bib.bib200)\)\.As a corollary of Theorem[1](https://arxiv.org/html/2605.21692#Thmtheorem1), Eq\.[2](https://arxiv.org/html/2605.21692#S3.E2)naturally defines an estimator of the intrinsic dimension of the data manifoldΩ\\Omega\. Using the experimental settings described above, we estimate the intrinsic dimension of several synthetic datasets and report the results in Table[1](https://arxiv.org/html/2605.21692#S5.T1)\. The estimated dimensions closely match the ground\-truth manifold dimensions across different geometries and ambient dimensions, which empirically validates the estimator in controlled settings\.

We further evaluate the estimator on real\-world datasets usingi\.i\.d\.sampling\. As shown in Table[1](https://arxiv.org/html/2605.21692#S5.T1), our estimates are consistent with prior intrinsic dimension estimators\(Popeet al\.,[2021](https://arxiv.org/html/2605.21692#bib.bib195); Ansuiniet al\.,[2019](https://arxiv.org/html/2605.21692#bib.bib200)\)\. We observe a progressive increase in intrinsic dimension with dataset complexity: MNIST and SVHN, which both correspond to digit recognition tasks, exhibit similar dimensions, while CIFAR\-10 has a larger estimated dimension consistent with its greater visual diversity\. MSCOCO and Tiny ImageNet, which contain a broader range of semantic classes and visual structures, exhibit the highest intrinsic dimensions\.

## 6Conclusion

This work introduces a new metric – the representation gap –, that characterizes neural network generalization from a geometric perspective\. We provide a detailed asymptotic analysis of this representation gap in two important settings: non\-conditional generative modeling and supervised prediction, under bothi\.i\.d\.and optimally diverse sampling regimes\. We show that the asymptotic scaling of the representation gap is governed by a single parameter, the intrinsic dimension of the task, and relates naturally to classical notions of generalization\. In particular, we show how standard machine learning techniques, such as equivariant architectures, reduce this intrinsic dimension, thereby provably improving asymptotic generalization\. We validate our theoretical results both on controlled synthetic environments and real\-world datasets\. Our results suggest that intrinsic dimension may serve as a unifying geometric principle for understanding generalization, and could be leveraged to inform network architecture and training pipeline design in a principled manner\.

More generally, our work suggests shifting the focus from the parameter space of neural networks and the statistical properties of training algorithms to the geometry of the data and prediction spaces\. Indeed, global properties of trained models – such as memorization, equivariance, or minimal\-norm interpolation – induce geometric structure in the prediction space, thereby reducing generalization to a geometric problem\. Beyond generalization itself, this perspective provides a principled way to characterize the information contained within a neural network through the geometry of its prediction space\. Important applications include architecture\-agnostic knowledge transfer and collaborative learning\. More fundamentally, we believe that the geometry of the prediction space may provide a principled notion of information based on teachability, opening the way toward a geometric understanding of real\-world task uncertainty and the intrinsic limits of statistical learning\.

## References

- A convergence theory for deep learning via over\-parameterization\.InInternational conference on machine learning,pp\. 242–252\.Cited by:[§1](https://arxiv.org/html/2605.21692#S1.p1.1)\.
- D\. Aloise, A\. Deshpande, P\. Hansen, and P\. Popat \(2009\)NP\-hardness of euclidean sum\-of\-squares clustering\.Machine learning75\(2\),pp\. 245–248\.Cited by:[§5](https://arxiv.org/html/2605.21692#S5.p11.1)\.
- A\. Ansuini, A\. Laio, J\. H\. Macke, and D\. Zoccolan \(2019\)Intrinsic dimension of data representations in deep neural networks\.Advances in Neural Information Processing Systems32\.Cited by:[§1](https://arxiv.org/html/2605.21692#S1.p2.1),[§2](https://arxiv.org/html/2605.21692#S2.p1.1),[§5\.2](https://arxiv.org/html/2605.21692#S5.SS2.p2.1),[Table 1](https://arxiv.org/html/2605.21692#S5.T1),[Table 1](https://arxiv.org/html/2605.21692#S5.T1.4.2),[§5](https://arxiv.org/html/2605.21692#S5.p7.1)\.
- D\. Arthur and S\. Vassilvitskii \(2006\)K\-means\+\+: the advantages of careful seeding\.Technical reportStanford\.Cited by:[§5](https://arxiv.org/html/2605.21692#S5.p11.1)\.
- D\. Barilari, I\. Beschastnyi, and A\. Lerario \(2018\)Volume of small balls and sub\-riemannian curvature in 3d contact manifolds\.arXiv preprint arXiv:1802\.10155\.Cited by:[§C\.3](https://arxiv.org/html/2605.21692#A3.SS3.4.p3.11)\.
- M\. Belkin \(2021\)Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2105.14368),2105\.14368Cited by:[§1](https://arxiv.org/html/2605.21692#S1.p1.1)\.
- Y\. Bengio, A\. Courville, and P\. Vincent \(2013\)Representation learning: a review and new perspectives\.IEEE Transactions on Pattern Analysis and Machine Intelligence35\(8\),pp\. 1798–1828\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2013.50),ISSN 1939\-3539,[Link](https://ieeexplore.ieee.org/document/6472238/)Cited by:[§A\.1](https://arxiv.org/html/2605.21692#A1.SS1.p1.13),[§2](https://arxiv.org/html/2605.21692#S2.p1.1),[§4\.1](https://arxiv.org/html/2605.21692#S4.SS1.p1.11)\.
- G\. Biau and L\. Devroye \(2015\)Lectures on the nearest neighbor method\.Vol\.246,Springer\.Cited by:[§B\.2](https://arxiv.org/html/2605.21692#A2.SS2.p1.6),[§C\.3](https://arxiv.org/html/2605.21692#A3.SS3.2.p1.9),[§2](https://arxiv.org/html/2605.21692#S2.p4.1)\.
- M\. M\. Bronstein, J\. Bruna, T\. Cohen, and P\. Veličković \(2021\)Geometric deep learning: grids, groups, graphs, geodesics, and gauges\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2104.13478),2104\.13478Cited by:[§C\.4](https://arxiv.org/html/2605.21692#A3.SS4.p2.1)\.
- S\. Bulusu, M\. Favoni, A\. Ipp, D\. I\. Müller, and D\. Schuh \(2022\)Equivariance and generalization in neural networks\.EPJ Web of Conferences258,pp\. 09001\.External Links:[Document](https://dx.doi.org/10.1051/epjconf/202225809001),ISSN 2100\-014XCited by:[§2](https://arxiv.org/html/2605.21692#S2.p2.2)\.
- S\. Chen, E\. Dobriban, and J\. H\. Lee \(2020\)A group\-theoretic framework for data augmentation\.Journal of Machine Learning Research\.Cited by:[§1](https://arxiv.org/html/2605.21692#S1.p2.1),[§2](https://arxiv.org/html/2605.21692#S2.p2.2)\.
- P\. Chiang, R\. Ni, D\. Y\. Miller, A\. Bansal, J\. Geiping, M\. Goldblum, and T\. Goldstein \(2022\)Loss landscapes are all you need: neural network generalization can be explained without the implicit bias of gradient descent\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=QC10RmRbZy9)Cited by:[§1](https://arxiv.org/html/2605.21692#S1.p1.1)\.
- T\. Cohen and M\. Welling \(2016\)Group equivariant convolutional networks\.InInternational conference on machine learning,pp\. 2990–2999\.Cited by:[§2](https://arxiv.org/html/2605.21692#S2.p2.2)\.
- J\. A\. Costa and A\. O\. Hero III \(2006\)Determining intrinsic dimension and entropy of high\-dimensional shape spaces\.InStatistics and analysis of shapes,pp\. 231–252\.Cited by:[§B\.2](https://arxiv.org/html/2605.21692#A2.SS2.p1.6)\.
- J\. Deng, W\. Dong, R\. Socher, L\. Li, K\. Li, and L\. Fei\-Fei \(2009\)Imagenet: a large\-scale hierarchical image database\.In2009 IEEE conference on computer vision and pattern recognition,pp\. 248–255\.Cited by:[§1](https://arxiv.org/html/2605.21692#S1.p4.1)\.
- R\. Durrett \(2019\)Probability: theory and examples\.Vol\.49,Cambridge university press\.Cited by:[§C\.3](https://arxiv.org/html/2605.21692#A3.SS3.5.p4.3)\.
- B\. Elesedy and S\. Zaidi \(2021\)Provably strict generalisation benefit for equivariant models\.InInternational conference on machine learning,pp\. 2959–2969\.Cited by:[§1](https://arxiv.org/html/2605.21692#S1.p2.1),[§2](https://arxiv.org/html/2605.21692#S2.p2.2)\.
- C\. Fefferman, S\. Mitter, and H\. Narayanan \(2016\)Testing the manifold hypothesis\.Journal of the American Mathematical Society29\(4\),pp\. 983–1049\.External Links:[Document](https://dx.doi.org/10.1090/jams/852),ISSN 0894\-0347, 1088\-6834,[Link](https://www.ams.org/jams/2016-29-04/S0894-0347-2016-00852-4/)Cited by:[§1](https://arxiv.org/html/2605.21692#S1.p1.1)\.
- E\. Finn, T\. A\. Keller, M\. Theodosis, and D\. E\. Ba \(2025\)Origins of creativity in attention\-based diffusion models\.InHiLD at ICML 2025,External Links:[Link](http://arxiv.org/abs/2506.17324),[Document](https://dx.doi.org/10.48550/arXiv.2506.17324),2506\.17324Cited by:[§1](https://arxiv.org/html/2605.21692#S1.p1.1),[§4\.1](https://arxiv.org/html/2605.21692#S4.SS1.p2.7)\.
- S\. Gallot, D\. Hulin, and J\. Lafontaine \(1990\)Riemannian geometry\.2nd ed edition,Universitext,Springer\-Verlag,Berlin ; New York\.External Links:ISBN 978\-3\-540\-52401\-4 978\-0\-387\-52401\-6Cited by:[§C\.4](https://arxiv.org/html/2605.21692#A3.SS4.10.p1.2),[§4\.3](https://arxiv.org/html/2605.21692#S4.SS3.1.p1.3)\.
- H\. Georgi \(2000\)Lie algebras in particle physics: from isospin to unified theories\.Taylor & Francis\.Cited by:[§C\.4](https://arxiv.org/html/2605.21692#A3.SS4.p2.1)\.
- R\. Gilmore \(2006\)Lie groups, lie algebras, and some of their applications\.Courier Corporation\.Note:Google\-Books\-ID: N8UsAwAAQBAJExternal Links:ISBN 978\-0\-486\-44529\-8Cited by:[§C\.4](https://arxiv.org/html/2605.21692#A3.SS4.p2.1)\.
- S\. Gong, V\. N\. Boddeti, and A\. K\. Jain \(2019\)On the intrinsic dimensionality of image representations\.In2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 3982–3991\.External Links:ISBN 978\-1\-7281\-3293\-8,[Link](https://ieeexplore.ieee.org/document/8953348/),[Document](https://dx.doi.org/10.1109/CVPR.2019.00411)Cited by:[§1](https://arxiv.org/html/2605.21692#S1.p2.1),[§2](https://arxiv.org/html/2605.21692#S2.p1.1)\.
- I\. J\. Goodfellow, J\. Pouget\-Abadie, M\. Mirza, B\. Xu, D\. Warde\-Farley, S\. Ozair, A\. Courville, and Y\. Bengio \(2014\)Generative adversarial networks\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1406.2661),1406\.2661Cited by:[§C\.1](https://arxiv.org/html/2605.21692#A3.SS1.p2.1)\.
- S\. Graf, H\. Luschgy, and G\. Pagès \(2008\)Distortion mismatch in the quantization of probability measures\.ESAIM: Probability and Statistics12,pp\. 127–153\.Cited by:[§B\.1](https://arxiv.org/html/2605.21692#A2.SS1.p3.1)\.
- S\. Graf and H\. Luschgy \(2007\)Foundations of quantization for probability distributions\.Springer\.Cited by:[§B\.1](https://arxiv.org/html/2605.21692#A2.SS1.p2.2),[§B\.1](https://arxiv.org/html/2605.21692#A2.SS1.p4.5),[§C\.1](https://arxiv.org/html/2605.21692#A3.SS1.p6.2),[§3](https://arxiv.org/html/2605.21692#S3.p5.3)\.
- P\. M\. Gruber \(2001\)Optimal configurations of finite sets in riemannian 2\-manifolds\.Geometriae Dedicata84\(1\),pp\. 271–320\.Cited by:[§B\.1](https://arxiv.org/html/2605.21692#A2.SS1.p5.1),[§C\.3](https://arxiv.org/html/2605.21692#A3.SS3.1.p1.2),[§C\.3](https://arxiv.org/html/2605.21692#A3.SS3.p2.9),[§2](https://arxiv.org/html/2605.21692#S2.p4.1),[§4\.3](https://arxiv.org/html/2605.21692#S4.SS3.1.p1.4)\.
- J\. Ho, A\. Jain, and P\. Abbeel \(2020\)Denoising diffusion probabilistic models\.Advances in neural information processing systems33,pp\. 6840–6851\.Cited by:[§C\.1](https://arxiv.org/html/2605.21692#A3.SS1.p2.1)\.
- E\. Hoogeboom, V\. G\. Satorras, C\. Vignac, and M\. Welling \(2022\)Equivariant diffusion for molecule generation in 3d\.InInternational conference on machine learning,pp\. 8867–8887\.Cited by:[§3](https://arxiv.org/html/2605.21692#S3.p2.6)\.
- K\. Hornik \(1991\)Approximation capabilities of multilayer feedforward networks\.Neural Networks4\(2\),pp\. 251–257\.External Links:[Document](https://dx.doi.org/10.1016/0893-6080%2891%2990009-T),ISSN 0893\-6080,[Link](https://linkinghub.elsevier.com/retrieve/pii/089360809190009T)Cited by:[§1](https://arxiv.org/html/2605.21692#S1.p1.1)\.
- M\. Iacobelli \(2016\)Asymptotic quantization for probability measures on riemannian manifolds\.ESAIM: Control, Optimisation and Calculus of Variations22\(3\),pp\. 770–785\.Cited by:[§B\.1](https://arxiv.org/html/2605.21692#A2.SS1.p3.1)\.
- A\. K\. Jacobsen, J\. M\. Gegenfurtner, and G\. Arvanitidis \(2025\)Staying on the manifold: geometry\-aware noise injection\.arXiv preprint arXiv:2509\.20201\.Cited by:[§5](https://arxiv.org/html/2605.21692#S5.p6.1)\.
- M\. Kamb and S\. Ganguli \(2025\)An analytic theory of creativity in convolutional diffusion models\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2412.20292),2412\.20292Cited by:[§C\.4](https://arxiv.org/html/2605.21692#A3.SS4.6.p1.1),[§C\.4](https://arxiv.org/html/2605.21692#A3.SS4.p1.5),[§C\.4](https://arxiv.org/html/2605.21692#A3.SS4.p2.1),[§C\.4](https://arxiv.org/html/2605.21692#A3.SS4.p4.10),[§1](https://arxiv.org/html/2605.21692#S1.p1.1),[§2](https://arxiv.org/html/2605.21692#S2.p2.2),[§4\.1](https://arxiv.org/html/2605.21692#S4.SS1.p2.7),[§4\.2](https://arxiv.org/html/2605.21692#S4.SS2.p2.5)\.
- J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei \(2020a\)Scaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.Cited by:[§2](https://arxiv.org/html/2605.21692#S2.p3.1)\.
- J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei \(2020b\)Scaling laws for neural language models\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2001.08361),2001\.08361Cited by:[§1](https://arxiv.org/html/2605.21692#S1.p1.1)\.
- D\. P\. Kingma and J\. Ba \(2017\)Adam: a method for stochastic optimization\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1412.6980),1412\.6980Cited by:[§5](https://arxiv.org/html/2605.21692#S5.p9.4)\.
- D\. P\. Kingma and M\. Welling \(2022\)Auto\-encoding variational bayes\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1312.6114),1312\.6114Cited by:[§C\.1](https://arxiv.org/html/2605.21692#A3.SS1.p2.1)\.
- W\. D\. Kirwin \(2010\)Higher asymptotics of laplace’s approximation\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.0810.1700),0810\.1700Cited by:[§C\.4](https://arxiv.org/html/2605.21692#A3.SS4.4.p4.11)\.
- A\. Krizhevsky and G\. Hinton \(2009\)Learning multiple layers of features from tiny images\.Technical reportUniversity of Toronto\.Cited by:[§5](https://arxiv.org/html/2605.21692#S5.p7.1)\.
- M\. Kubo, R\. Banno, H\. Manabe, and M\. Minoji \(2019\)Implicit regularization in over\-parameterized neural networks\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1903.01997),1903\.01997Cited by:[§1](https://arxiv.org/html/2605.21692#S1.p1.1)\.
- Y\. Le, X\. Yang,et al\.\(2015\)Tiny imagenet visual recognition challenge\.CS 231N7\(7\),pp\. 3\.Cited by:[§5](https://arxiv.org/html/2605.21692#S5.p7.1)\.
- Y\. Lecun, L\. Bottou, Y\. Bengio, and P\. Haffner \(1998\)Gradient\-based learning applied to document recognition\.Proceedings of the IEEE86\(11\),pp\. 2278–2324\.External Links:[Document](https://dx.doi.org/10.1109/5.726791),ISSN 1558\-2256,[Link](https://ieeexplore.ieee.org/document/726791/)Cited by:[§5](https://arxiv.org/html/2605.21692#S5.p7.1)\.
- J\. M\. Lee \(2006\)Riemannian manifolds: an introduction to curvature\.Springer Science & Business Media\.Note:Google\-Books\-ID: 92PgBwAAQBAJExternal Links:ISBN 978\-0\-387\-22726\-9Cited by:[§A\.1](https://arxiv.org/html/2605.21692#A1.SS1.p1.13),[§A\.1](https://arxiv.org/html/2605.21692#A1.SS1.p2.12),[§A\.1](https://arxiv.org/html/2605.21692#A1.SS1.p3.14)\.
- H\. Li, Y\. Zou, Y\. Wang, O\. Majumder, Y\. Xie, R\. Manmatha, A\. Swaminathan, Z\. Tu, S\. Ermon, and S\. Soatto \(2024a\)On the scalability of diffusion\-based text\-to\-image generation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 9400–9409\.Cited by:[§2](https://arxiv.org/html/2605.21692#S2.p3.1)\.
- X\. Li, Q\. Zhang, D\. Kang, W\. Cheng, Y\. Gao, J\. Zhang, Z\. Liang, J\. Liao, Y\. Cao, and Y\. Shan \(2024b\)Advances in 3d generation: a survey\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2401.17807),2401\.17807Cited by:[§3](https://arxiv.org/html/2605.21692#S3.p1.6)\.
- Z\. Liang, H\. He, C\. Yang, and B\. Dai \(2024\)Scaling laws for diffusion transformers\.arXiv preprint arXiv:2410\.08184\.Cited by:[§2](https://arxiv.org/html/2605.21692#S2.p3.1)\.
- T\. Lin, M\. Maire, S\. Belongie, J\. Hays, P\. Perona, D\. Ramanan, P\. Dollár, and C\. L\. Zitnick \(2014a\)Fleet d, pajdla t, schiele b, tuytelaars t, et al\. microsoft coco: common objects in context\.Computer Vision–ECCV 2014,pp\. 740–755\.Cited by:[§1](https://arxiv.org/html/2605.21692#S1.p4.1)\.
- T\. Lin, M\. Maire, S\. Belongie, J\. Hays, P\. Perona, D\. Ramanan, P\. Dollár, and C\. L\. Zitnick \(2014b\)Microsoft coco: common objects in context\.InEuropean conference on computer vision,pp\. 740–755\.Cited by:[§5](https://arxiv.org/html/2605.21692#S5.p7.1)\.
- S\. Lloyd \(1982\)Least squares quantization in pcm\.IEEE transactions on information theory28\(2\),pp\. 129–137\.Cited by:[§5](https://arxiv.org/html/2605.21692#S5.p11.1)\.
- G\. Loaiza\-Ganem, B\. L\. Ross, R\. Hosseinzadeh, A\. L\. Caterini, and J\. C\. Cresswell \(2024\)Deep generative models through the lens of the manifold hypothesis: a survey and new connections\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2404.02954),2404\.02954Cited by:[§2](https://arxiv.org/html/2605.21692#S2.p1.1)\.
- K\. Mei, Z\. Tu, M\. Delbracio, H\. Talebi, V\. M\. Patel, and P\. Milanfar \(2024\)Bigger is not always better: scaling properties of latent diffusion models\.Transactions on Machine Learning Research\.Cited by:[§2](https://arxiv.org/html/2605.21692#S2.p3.1)\.
- M\. Mohri and A\. Rostamizadeh \(2008\)Rademacher complexity bounds for non\-iid processes\.Advances in neural information processing systems21\.Cited by:[§C\.1](https://arxiv.org/html/2605.21692#A3.SS1.p5.1)\.
- Y\. Netzer, T\. Wang, A\. Coates, A\. Bissacco, B\. Wu, and A\. Y\. Ng \(2011\)Reading digits in natural images with unsupervised feature learning\.InNIPS Workshop on Deep Learning and Unsupervised Feature Learning,Granada, Spain\.Cited by:[§5](https://arxiv.org/html/2605.21692#S5.p7.1)\.
- D\. Newman \(1982\)The hexagon theorem\.IEEE Transactions on information theory28\(2\),pp\. 137–139\.Cited by:[§B\.1](https://arxiv.org/html/2605.21692#A2.SS1.p4.5)\.
- G\. Pagès and J\. Printems \(2003\)Optimal quadratic quantization for numerics: the gaussian case\.Monte Carlo Methods Appl\.9\(2\),pp\. 135–165\.Cited by:[§B\.1](https://arxiv.org/html/2605.21692#A2.SS1.p4.5)\.
- M\. D\. Penrose and J\. E\. Yukich \(2013\)Limit theory for point processes in manifolds\.The Annals of Applied Probability,pp\. 2161–2211\.Cited by:[§2](https://arxiv.org/html/2605.21692#S2.p4.1)\.
- G\. Peyré, M\. Cuturi,et al\.\(2019\)Computational optimal transport: with applications to data science\.Foundations and Trends® in Machine Learning11\(5\-6\),pp\. 355–607\.Cited by:[§A\.1](https://arxiv.org/html/2605.21692#A1.SS1.p3.14),[§B\.1](https://arxiv.org/html/2605.21692#A2.SS1.p2.2),[§E\.2](https://arxiv.org/html/2605.21692#A5.SS2.p1.5),[§3](https://arxiv.org/html/2605.21692#S3.p5.3)\.
- P\. Pope, C\. Zhu, A\. Abdelkader, M\. Goldblum, and T\. Goldstein \(2021\)The intrinsic dimension of images and its impact on learning\.arXiv\.External Links:[Link](http://arxiv.org/abs/2104.08894),[Document](https://dx.doi.org/10.48550/arXiv.2104.08894),2104\.08894 \[cs\]Cited by:[§2](https://arxiv.org/html/2605.21692#S2.p1.1),[§5\.2](https://arxiv.org/html/2605.21692#S5.SS2.p2.1),[Table 1](https://arxiv.org/html/2605.21692#S5.T1),[Table 1](https://arxiv.org/html/2605.21692#S5.T1.4.2),[§5](https://arxiv.org/html/2605.21692#S5.p7.1)\.
- P\. Ramachandran, B\. Zoph, and Q\. V\. Le \(2017\)Searching for activation functions\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1710.05941),1710\.05941Cited by:[§5](https://arxiv.org/html/2605.21692#S5.p8.1)\.
- D\. J\. Rezende and S\. Mohamed \(2016\)Variational inference with normalizing flows\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1505.05770),1505\.05770Cited by:[§C\.1](https://arxiv.org/html/2605.21692#A3.SS1.p2.1)\.
- D\. E\. Rumelhart, G\. E\. Hinton, and R\. J\. Williams \(1986\)Learning representations by back\-propagating errors\.nature323\(6088\),pp\. 533–536\.Cited by:[§5](https://arxiv.org/html/2605.21692#S5.p8.1)\.
- A\. Sannai, M\. Imaizumi, and M\. Kawano \(2021\)Improved generalization bounds of group invariant/equivariant deep networks via quotient feature spaces\.InUncertainty in artificial intelligence,pp\. 771–780\.Cited by:[§2](https://arxiv.org/html/2605.21692#S2.p2.2)\.
- V\. Schuster and A\. Krogh \(2021\)A manifold learning perspective on representation learning: learning decoder and representations without an encoder\.Entropy23\(11\),pp\. 1403\.Note:Publisher: Multidisciplinary Digital Publishing InstituteExternal Links:[Document](https://dx.doi.org/10.3390/e23111403),ISSN 1099\-4300,[Link](https://www.mdpi.com/1099-4300/23/11/1403)Cited by:[§2](https://arxiv.org/html/2605.21692#S2.p1.1)\.
- S\. Shalev\-Shwartz and S\. Ben\-David \(2014\)Understanding machine learning: from theory to algorithms\.Cambridge university press\.Cited by:[§C\.1](https://arxiv.org/html/2605.21692#A3.SS1.p5.1),[§E\.1](https://arxiv.org/html/2605.21692#A5.SS1.p1.2),[§1](https://arxiv.org/html/2605.21692#S1.p4.1),[§4\.5](https://arxiv.org/html/2605.21692#S4.SS5.p1.2)\.
- J\. Simon, D\. Kunin, A\. Atanasov, E\. Boix\-Adserà, B\. Bordelon, J\. Cohen, N\. Ghosh, F\. Guth, A\. Jacot, M\. Kamb,et al\.\(2026\)There will be a scientific theory of deep learning\.arXiv preprint arXiv:2604\.21691\.Cited by:[§1](https://arxiv.org/html/2605.21692#S1.p1.1)\.
- J\. Song, C\. Meng, and S\. Ermon \(2022\)Denoising diffusion implicit models\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2010.02502),2010\.02502Cited by:[§A\.4](https://arxiv.org/html/2605.21692#A1.SS4.p1.7),[§C\.1](https://arxiv.org/html/2605.21692#A3.SS1.p2.1),[§4\.1](https://arxiv.org/html/2605.21692#S4.SS1.p2.7),[§5](https://arxiv.org/html/2605.21692#S5.p9.4)\.
- Y\. Song and S\. Ermon \(2019\)Generative modeling by estimating gradients of the data distribution\.Advances in neural information processing systems32\.Cited by:[§C\.1](https://arxiv.org/html/2605.21692#A3.SS1.p2.1),[§4\.2](https://arxiv.org/html/2605.21692#S4.SS2.p1.8)\.
- B\. Tahmasebi and S\. Jegelka \(2023\)The exact sample complexity gain from invariances for kernel regression\.InNeural Information Processing Systems,Cited by:[§E\.1](https://arxiv.org/html/2605.21692#A5.SS1.p2.5),[§1](https://arxiv.org/html/2605.21692#S1.p2.1),[§2](https://arxiv.org/html/2605.21692#S2.p2.2),[§4\.5](https://arxiv.org/html/2605.21692#S4.SS5.p2.5)\.
- D\. Teney, A\. M\. Nicolicioiu, V\. Hartmann, and E\. Abbasnejad \(2024\)Neural redshift: random networks are not random functions\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 4786–4796\.External Links:ISBN 979\-8\-3503\-5300\-6,[Link](https://ieeexplore.ieee.org/document/10658325/),[Document](https://dx.doi.org/10.1109/CVPR52733.2024.00458)Cited by:[§1](https://arxiv.org/html/2605.21692#S1.p1.1)\.
- L\. Theis, A\. v\. d\. Oord, and M\. Bethge \(2015\)A note on the evaluation of generative models\.arXiv preprint arXiv:1511\.01844\.Cited by:[§E\.2](https://arxiv.org/html/2605.21692#A5.SS2.p1.5),[§1](https://arxiv.org/html/2605.21692#S1.p2.1)\.
- A\. Torralba and A\. A\. Efros \(2011\)Unbiased look at dataset bias\.InCVPR 2011,pp\. 1521–1528\.Cited by:[§C\.1](https://arxiv.org/html/2605.21692#A3.SS1.p5.1),[§1](https://arxiv.org/html/2605.21692#S1.p4.1),[§4\.3](https://arxiv.org/html/2605.21692#S4.SS3.p2.7)\.
- G\. Yang, X\. Huang, Z\. Hao, M\. Liu, S\. Belongie, and B\. Hariharan \(2019\)PointFlow: 3d point cloud generation with continuous normalizing flows\.In2019 IEEE/CVF International Conference on Computer Vision \(ICCV\),Seoul, Korea \(South\),pp\. 4540–4549\.External Links:[Document](https://dx.doi.org/10.1109/ICCV.2019.00464),ISBN 978\-1\-7281\-4803\-8,[Link](https://ieeexplore.ieee.org/document/9010395/)Cited by:[§3](https://arxiv.org/html/2605.21692#S3.p1.6)\.
- J\. Yao, M\. Goswami, and C\. Chen \(2024\)A theoretical study of neural network expressive power via manifold topology\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2410.16542),2410\.16542Cited by:[§2](https://arxiv.org/html/2605.21692#S2.p1.1)\.
- P\. Zador \(1982\)Asymptotic quantization error of continuous signals and the quantization dimension\.IEEE Transactions on Information Theory28\(2\),pp\. 139–149\.Cited by:[§B\.1](https://arxiv.org/html/2605.21692#A2.SS1.p3.1),[§C\.1](https://arxiv.org/html/2605.21692#A3.SS1.p3.6),[§1](https://arxiv.org/html/2605.21692#S1.p4.1)\.
- C\. Zhang, S\. Bengio, M\. Hardt, B\. Recht, and O\. Vinyals \(2021\)Understanding deep learning \(still\) requires rethinking generalization\.Communications of the ACM64\(3\),pp\. 107–115\.External Links:[Document](https://dx.doi.org/10.1145/3446776),ISSN 0001\-0782, 1557\-7317Cited by:[§1](https://arxiv.org/html/2605.21692#S1.p1.1),[§4\.3](https://arxiv.org/html/2605.21692#S4.SS3.p4.3)\.

## Appendix ANotations

### A\.1Task and geometry

Manifold\.We consider a supervised task, with input space𝒳⊂ℛd𝒳\\mathcal\{X\}\\subset\\mathcal\{R\}^\{d\_\{\\mathcal\{X\}\}\}and target space𝒴⊂ℛd𝒴\\mathcal\{Y\}\\subset\\mathcal\{R\}^\{d\_\{\\mathcal\{Y\}\}\}\. We assume that the observations\(x,y\)\(x,y\)belong to a subsetΩ⊂𝒳×𝒴\\Omega\\subset\\mathcal\{X\}\\times\\mathcal\{Y\}, which models the structure of the task and its underlying symmetries\. Following the manifold hypothesis\(Bengioet al\.,[2013](https://arxiv.org/html/2605.21692#bib.bib135)\), we assume thatΩ\\Omegacorresponds to a low\-dimensional manifold embedded in the ambient space𝒳×𝒴\\mathcal\{X\}\\times\\mathcal\{Y\}\. More precisely, we assume throughout this work thatΩ\\Omegais a Riemannian manifold\(Lee,[2006](https://arxiv.org/html/2605.21692#bib.bib160)\), and we denote bydΩd\_\{\\Omega\}its dimension\. We denote byΩ𝒳\\Omega\_\{\\mathcal\{X\}\}andΩ𝒴\\Omega\_\{\\mathcal\{Y\}\}the projections ofΩ\\Omegaonto𝒳\\mathcal\{X\}and𝒴\\mathcal\{Y\}respectively\.

Quotient manifold\.Machine learning tasks typically feature symmetries \(e\.g\., translation invariance in image classification or rotational equivariance in molecular modeling\), which are reflected in the structure of the manifoldΩ\\Omega\. In this work, we focus on tasks exhibiting symmetries described by a groupGGacting on the manifoldΩ\\Omega\. We denote byG​\(y\)=\{g​\(y\)∣g∈G\}G\(y\)=\\\{g\(y\)\\mid g\\in G\\\}the orbit of a pointy∈Ωy\\in\\Omegaunder the action ofGG, and byG​\(E\)=∪y∈EG​\(y\)G\(E\)=\\cup\_\{y\\in E\}G\(y\)the orbit of a setEE\. We will assume thatGGis a Lie group acting by isometries onΩ\\Omega\. Under this assumption, we can define the quotient manifoldΩ/G\\Omega/G\(Lee,[2006](https://arxiv.org/html/2605.21692#bib.bib160)\), and denote bydΩ/Gd\_\{\\Omega/G\}its dimension\.

Metric\.We denote byℓ:Ω×Ω→ℝ\+\\ell:\\Omega\\times\\Omega\\to\\mathbb\{R\}\_\{\+\}a non\-negative cost function onΩ\\Omega\. Unless stated otherwise,ℓ\\ellcorresponds to the squared geodesic distance induced by the Riemannian metric onΩ\\Omega\(Peyréet al\.,[2019](https://arxiv.org/html/2605.21692#bib.bib211)\)\. We denote byℓ𝒳\\ell\_\{\\mathcal\{X\}\}andℓ𝒴\\ell\_\{\\mathcal\{Y\}\}the corresponding cost functions induced by projection onto𝒳\\mathcal\{X\}and𝒴\\mathcal\{Y\}\. For a pointy∈Ωy\\in\\Omegaand a subsetE⊂ΩE\\subset\\Omega, we defineℓ​\(y,E\)=infy′∈Eℓ​\(y,y′\)\\ell\(y,E\)=\\inf\_\{y^\{\\prime\}\\in E\}\\ell\(y,y^\{\\prime\}\)\. We denote byℓΩ/G\\ell\_\{\\Omega/G\}the quotient metric induced byℓ\\ellonΩ/G\\Omega/G\(Lee,[2006](https://arxiv.org/html/2605.21692#bib.bib160)\)\.

### A\.2Data and model

Data\.We assume access to a dataset𝔻⊂Ω\\mathbb\{D\}\\subset\\Omegacomposed ofnnobservations\. For a given inputx∈𝒳x\\in\\mathcal\{X\}, we define the conditional manifoldΩx=\{y∣\(x,y\)∈Ω\}\\Omega\_\{x\}=\\\{y\\mid\(x,y\)\\in\\Omega\\\}, which corresponds to the set of admissible targets associated with the inputxx\. Similarly, we define the conditional dataset𝔻x=\{y∈𝔻𝒴∣\(x,y\)∈𝔻\}\\mathbb\{D\}\_\{x\}=\\\{y\\in\\mathbb\{D\}\_\{\\mathcal\{Y\}\}\\mid\(x,y\)\\in\\mathbb\{D\}\\\}\. We denote by𝔻𝒳=\{x∣\(x,y\)∈𝔻\}\\mathbb\{D\}\_\{\\mathcal\{X\}\}=\\\{x\\mid\(x,y\)\\in\\mathbb\{D\}\\\}and𝔻𝒴=\{y∣\(x,y\)∈𝔻\}\\mathbb\{D\}\_\{\\mathcal\{Y\}\}=\\\{y\\mid\(x,y\)\\in\\mathbb\{D\}\\\}the sets of inputs and targets appearing in𝔻\\mathbb\{D\}respectively\. In deterministic supervised settings, each inputx∈Ω𝒳x\\in\\Omega\_\{\\mathcal\{X\}\}is associated with a unique targety​\(x\)∈Ω𝒴y\(x\)\\in\\Omega\_\{\\mathcal\{Y\}\}, so that the manifoldΩ\\Omegacan be identified with the graph of a functiony:Ω𝒳→Ω𝒴y:\\Omega\_\{\\mathcal\{X\}\}\\to\\Omega\_\{\\mathcal\{Y\}\}\.

Model\.We consider neural networksfθf\_\{\\theta\}belonging to a parametric familyℱΘ⊂ℱ​\(𝒳,𝒴\)\\mathcal\{F\}\_\{\\Theta\}\\subset\\mathcal\{F\}\(\\mathcal\{X\},\\mathcal\{Y\}\)\. When there is no ambiguity, we will simplify the notation and denote the neural networks byff\. A modelffis said to be equivariant under the action of a groupGGif we haveg​\(f​\(x\)\)=f​\(g​\(x\)\)g\(f\(x\)\)=f\(g\(x\)\)for allx∈𝒳x\\in\\mathcal\{X\}andg∈Gg\\in G\.

### A\.3Probability and asymptotic notation

Probability\.We denote byℙ\\mathbb\{P\}a probability distribution supported onΩ\\Omega, and byppits density\. We denote byδx\\delta\_\{x\}the Dirac distribution centered at a pointxx\. We denote by𝒩​\(μ,σ2\)\\mathcal\{N\}\(\\mu,\\sigma^\{2\}\)the Gaussian distribution with meanμ\\muand varianceσ2\\sigma^\{2\}, and by𝒩​\(y∣μ,σ2\)\\mathcal\{N\}\(y\\mid\\mu,\\sigma^\{2\}\)the evaluation of its density at a pointyy\. We denote by𝟙​\[E\]\\mathds\{1\}\[E\]the indicator function of a setEE\. For a finite setEE, we denote by\|E\|\|E\|its cardinality\. IfEEis measurable,\|E\|\|E\|denotes its Lebesgue measure, andE̊\\mathring\{E\}its interior\.

Asymptotic notation\.We denote byan∼bna\_\{n\}\\sim b\_\{n\}the deterministic asymptotic equivalenceanbn→1\\frac\{a\_\{n\}\}\{b\_\{n\}\}\\to 1\. Similarly, we writeXn∼ℙanX\_\{n\}\\sim\_\{\\mathbb\{P\}\}a\_\{n\}whenXnan→1\\frac\{X\_\{n\}\}\{a\_\{n\}\}\\to 1in probability\. We use the standard notations→d\\to\_\{d\},→L1\\to\_\{L^\{1\}\}and→ℙ\\to\_\{\\mathbb\{P\}\}for convergence in distribution, convergence inL1L^\{1\}, and convergence in probability respectively\.

### A\.4Diffusion model

We will focus on Denoising Diffusion Implicit Models \(DDIM\) diffusion models\(Songet al\.,[2022](https://arxiv.org/html/2605.21692#bib.bib159)\)\. These models are trained to reverse a stochastic forward diffusion process that incrementally adds Gaussian noise to the data distribution while shrinking data points toward the origin\. Noise addition is governed by a noise scheduleαt\\alpha\_\{t\}, witht∈\[0,T\]t\\in\[0,T\]\. At each schedule step, the noised distribution can be writtenπt​\(y\)=1\|𝔻\|​∑z∈𝔻𝒩​\(y\|αt​z,\(1−αt\)​I\)\\pi\_\{t\}\(y\)=\\frac\{1\}\{\|\\mathbb\{D\}\|\}\\sum\_\{z\\in\\mathbb\{D\}\}\\mathcal\{N\}\(y\|\\sqrt\{\\alpha\_\{t\}\}z,\(1\-\\alpha\_\{t\}\)I\)\. In particular,π0=1\|𝔻\|​∑z∈𝔻δz\\pi\_\{0\}=\\frac\{1\}\{\|\\mathbb\{D\}\|\}\\sum\_\{z\\in\\mathbb\{D\}\}\\delta\_\{z\}recovers the empirical data distribution andπT=𝒩​\(0,I\)\\pi\_\{T\}=\\mathcal\{N\}\(0,I\)is an isotropic Gaussian distribution\. In this context, the modelfθ:𝒴×ℝ→𝒴f\_\{\\theta\}:\\mathcal\{Y\}\\times\\mathbb\{R\}\\rightarrow\\mathcal\{Y\}is trained to approximate the score functionst=∇log⁡πts\_\{t\}=\\nabla\\log\\pi\_\{t\}using the loss

ℒ​\(θ\)=𝔼t∼𝕌​\[0,T\],y0∼π0,η∼𝒩​\(0,I\)​‖fθ​\(αt​y0\+1−αt​η,t\)−η‖22\.\\mathcal\{L\}\(\\theta\)=\\mathbb\{E\}\_\{t\\sim\\mathbb\{U\}\[0,T\],y\_\{0\}\\sim\\pi\_\{0\},\\eta\\sim\\mathcal\{N\}\(0,I\)\}\\\|f\_\{\\theta\}\(\\sqrt\{\\alpha\_\{t\}\}y\_\{0\}\+\\sqrt\{1\-\\alpha\_\{t\}\}\\eta,t\)\-\\eta\\\|\_\{2\}^\{2\}\\;\.\(12\)At sampling time, an initial pointyT∼𝒩​\(0,I\)y\_\{T\}\\sim\\mathcal\{N\}\(0,I\)is sampled and then updated using the deterministic flow

y˙t=−γt​\(yt\+st​\(yt\)\),\\dot\{y\}\_\{t\}=\-\\gamma\_\{t\}\(y\_\{t\}\+s\_\{t\}\(y\_\{t\}\)\)\\;,\(13\)wherettgoes backward fromTTto0\. The output of the model corresponds to the endpoints of these trajectories\.

These equations can be generalized to the conditional case\. In particular, the modelfθ:𝒳×𝒴×ℝ→𝒴f\_\{\\theta\}:\\mathcal\{X\}\\times\\mathcal\{Y\}\\times\\mathbb\{R\}\\rightarrow\\mathcal\{Y\}is trained using the loss

ℒ​\(θ\)=𝔼t∼𝕌​\[0,T\],\(x0,y0\)∼π0,η∼𝒩​\(0,I\)​‖fθ​\(x0,αt​y0\+1−αt​η,t\)−η‖22\.\\mathcal\{L\}\(\\theta\)=\\mathbb\{E\}\_\{t\\sim\\mathbb\{U\}\[0,T\],\(x\_\{0\},y\_\{0\}\)\\sim\\pi\_\{0\},\\eta\\sim\\mathcal\{N\}\(0,I\)\}\\\|f\_\{\\theta\}\(x\_\{0\},\\sqrt\{\\alpha\_\{t\}\}y\_\{0\}\+\\sqrt\{1\-\\alpha\_\{t\}\}\\eta,t\)\-\\eta\\\|\_\{2\}^\{2\}\\;\.\(14\)

### A\.5Prediction space

We define the prediction spaceΩf\\Omega\_\{f\}of a modelffas follows\.

###### Definition 2\(Prediction space\)\.

Letffdenote a potentially non\-deterministic neural network\. For eachx∈𝒳x\\in\\mathcal\{X\}, we denote by𝒪f​\(x\)⊂𝒴\\mathcal\{O\}\_\{f\}\(x\)\\subset\\mathcal\{Y\}the set of outputs that can be generated byffwhen conditioned onxx\. We define the prediction space of the modelffby

Ωf=\{\(x,y\)\|x∈Ω𝒳,y∈𝒪f​\(x\)\}\.\\Omega\_\{f\}=\\\{\(x,y\)\\;\|\\;x\\in\\Omega\_\{\\mathcal\{X\}\},y\\in\\mathcal\{O\}\_\{f\}\(x\)\\;\\\}\\;\.\(15\)

In particular, ifffis a non\-conditional DDIM diffusion model,Ωf\\Omega\_\{f\}is the set of endpoints reachable by the reverse flow described by Eq\.[13](https://arxiv.org/html/2605.21692#A1.E13):

Ωf=\{y0\|yT∼𝒩​\(0,I\),yt​solves Eq\.[13](https://arxiv.org/html/2605.21692#A1.E13)\}\.\\Omega\_\{f\}=\\\{y\_\{0\}\\;\|\\;y\_\{T\}\\sim\\mathcal\{N\}\(0,I\)\\;,\\;y\_\{t\}\\text\{ solves Eq\. \}\\ref\{appeq:diffusion\_sampling\}\\\}\\;\\;\.\(16\)

## Appendix BPreliminaries

### B\.1Asymptotic quantization theory

A natural way to compare a manifoldΩ\\Omegawith a discrete approximation\{zk\}k∈\[\[1,n\]\]\\\{z\_\{k\}\\\}\_\{k\\in\[\\\!\[1,n\]\\\!\]\}is to use the quantization error

∫Ωmink∈\[\[1,n\]\]⁡ℓ​\(y,zk\)​p​\(y\)​𝑑y\.\\int\_\{\\Omega\}\\min\_\{k\\in\[\\\!\[1,n\]\\\!\]\}\\ell\(y,z\_\{k\}\)\\,p\(y\)\\,dy\.\(17\)
This quantity corresponds to a particular case of the Wasserstein distance\(Peyréet al\.,[2019](https://arxiv.org/html/2605.21692#bib.bib211)\)\(see Section[E\.2](https://arxiv.org/html/2605.21692#A5.SS2)\)\. Given a budget ofnnpoints, we are often interested in the best approximation achievable by a discrete set of sizenn\. This quantity, known as the optimal quantization error or optimal quantization risk\(Graf and Luschgy,[2007](https://arxiv.org/html/2605.21692#bib.bib182)\), is defined as

ℛn​\(ℙ\)=infz∈𝒴n∫𝒴mink∈\[\[1,n\]\]⁡ℓ​\(y,zk\)​p​\(y\)​𝑑y\.\\mathcal\{R\}\_\{n\}\(\\mathbb\{P\}\)=\\inf\_\{z\\in\\mathcal\{Y\}^\{n\}\}\\int\_\{\\mathcal\{Y\}\}\\min\_\{k\\in\[\\\!\[1,n\]\\\!\]\}\\ell\(y,z\_\{k\}\)\\,p\(y\)\\,dy\.\(18\)
A central tool of our analysis is Zador’s theorem\(Zador,[1982](https://arxiv.org/html/2605.21692#bib.bib179)\), a powerful result characterizing the asymptotic distribution of the centroids resulting from optimal quantization\. Intuitively, this theorem describes how well a continuous manifold can be approximated by a finite number of representative points, and how this approximation scales with the intrinsic dimension of the underlying space\. We first recall the Euclidean version \(seeGrafet al\.\([2008](https://arxiv.org/html/2605.21692#bib.bib178)\), Eq\. 2\.3, orIacobelli \([2016](https://arxiv.org/html/2605.21692#bib.bib180)\), Theorem 1\.3, for a more general version\)\.

###### Theorem 6\(Zador theorem\)\.

Letℙ=p​d​y\\mathbb\{P\}=p\\\>\\mathrm\{d\}ybe a Lebesgue\-dominated probability measure on a compact subset𝒴\\mathcal\{Y\}ofℝd\\mathbb\{R\}^\{d\}\. Then the optimal quantization errorℛn​\(ℙ\)\\mathcal\{R\}\_\{n\}\(\\mathbb\{P\}\)satisfies

ℛn​\(ℙ\)​∼n→∞​Jd∗​\(∫𝒴p​\(y\)dd\+2​dy\)d\+2d​1n2/d,\\mathcal\{R\}\_\{n\}\(\\mathbb\{P\}\)\\underset\{n\\to\\infty\}\{\\sim\}J\_\{d\}^\{\*\}\\;\\left\(\\int\_\{\\mathcal\{Y\}\}p\(y\)^\{\\frac\{d\}\{d\+2\}\}\\mathrm\{d\}y\\right\)^\{\\frac\{d\+2\}\{d\}\}\\frac\{1\}\{n^\{2/d\}\}\\\>,\(19\)whereJd∗J\_\{d\}^\{\*\}is the asymptotic optimal quantization error for the uniform distribution

Jd∗=infnn2/d​ℛn​\(𝒰​\(\[0,1\]d\)\)\.J\_\{d\}^\{\*\}=\\inf\_\{n\}n^\{2/d\}\\mathcal\{R\}\_\{n\}\(\\mathcal\{U\}\(\[0,1\]^\{d\}\)\)\\;\.\(20\)

The constantJd∗J\_\{d\}^\{\*\}can be computed for simple cases \(J1∗=112J^\{\*\}\_\{1\}=\\frac\{1\}\{12\}andJ2∗=518​3J^\{\*\}\_\{2\}=\\frac\{5\}\{18\\sqrt\{3\}\}\(Newman,[1982](https://arxiv.org/html/2605.21692#bib.bib183)\)\) and can be approximated for largeddbyJd∗∼d2​π​eJ^\{\*\}\_\{d\}\\sim\\frac\{d\}\{2\\pi e\}\(Pagès and Printems,[2003](https://arxiv.org/html/2605.21692#bib.bib181); Graf and Luschgy,[2007](https://arxiv.org/html/2605.21692#bib.bib182)\)\.

A generalization of Zador theorem to arbitrary manifolds has been proposed inGruber \([2001](https://arxiv.org/html/2605.21692#bib.bib92)\), which we report below \(see Theorem 2 in this reference for a stronger result\)\. In this case, the asymptotic behavior depends only on the intrinsic geometry of the manifold rather than on the ambient Euclidean space\.

###### Theorem 7\(Zador theorem on manifold\)\.

Letd=dΩd=d\_\{\\Omega\}denote the dimension of the manifoldΩ\\Omega\. Then there exists a constantJd∗J\_\{d\}^\{\*\}depending only onddand the metricℓ\\ellsuch that for allE⊂ΩE\\subset\\Omegacompact and measurable with\|E\|\>0\|E\|\>0and allp:E→ℝ\+p:E\\to\\mathbb\{R\}^\{\+\}continuous, we have

infz∈𝒴n∫Emink∈\[\[1,n\]\]⁡ℓ​\(y,zk\)​p​\(y\)​dy​∼n→∞​Jd∗​\(∫Ep​\(y\)dd\+2​dy\)d\+2d​1n2/d\.\\inf\_\{z\\in\\mathcal\{Y\}^\{n\}\}\\int\_\{E\}\\min\_\{k\\in\[\\\!\[1,n\]\\\!\]\}\\ell\(y,z\_\{k\}\)p\(y\)\\mathrm\{d\}y\\underset\{n\\to\\infty\}\{\\sim\}J\_\{d\}^\{\*\}\\;\\left\(\\int\_\{E\}p\(y\)^\{\\frac\{d\}\{d\+2\}\}\\mathrm\{d\}y\\right\)^\{\\frac\{d\+2\}\{d\}\}\\frac\{1\}\{n^\{2/d\}\}\\\>\.\(21\)

For an arbitrary manifoldΩ\\Omega, we define the density\-dependent volume functional appearing in Eq\.[19](https://arxiv.org/html/2605.21692#A2.E19)and Eq\.[21](https://arxiv.org/html/2605.21692#A2.E21)by

𝒱d∗​\(p\)=\(∫Ωp​\(y\)dd\+2​dy\)d\+2d\.\\mathcal\{V\}\_\{d\}^\{\*\}\(p\)=\\left\(\\int\_\{\\Omega\}p\(y\)^\{\\frac\{d\}\{d\+2\}\}\\mathrm\{d\}y\\right\)^\{\\frac\{d\+2\}\{d\}\}\\;\.\(22\)

### B\.2Point process theory

The quantization error in Eq\.[17](https://arxiv.org/html/2605.21692#A2.E17)is closely related to the nearest neighbor distance, which has been extensively studied by point process theory\(Biau and Devroye,[2015](https://arxiv.org/html/2605.21692#bib.bib7)\)\. Point process theory provides powerful tools to study the asymptotic geometry of random point configurations, including the nearest neighbor distancemink∈\[\[1,n\]\]⁡ℓ​\(Y,Zk\)\\min\_\{k\\in\[\\\!\[1,n\]\\\!\]\}\\ell\(Y,Z\_\{k\}\)whereYYandZkZ\_\{k\}are i\.i\.d variables following the data distributionℙ\\mathbb\{P\}\. An important result concerns the convergence rate of the expected nearest neighbor distance, which we state below in the Euclidean setting \(see Theorem 2\.3 p\.20 inBiau and Devroye \([2015](https://arxiv.org/html/2605.21692#bib.bib7)\)\)\. This result has also been extended to Riemannian manifoldsΩ\\Omegaand other metricsℓ\\ell\(see for instance Theorem 2 inCosta and Hero III \([2006](https://arxiv.org/html/2605.21692#bib.bib1)\)\)\.

###### Theorem 8\(Convergence rate of the nearest neighbor distance\)\.

AssumeΩ=\[0,1\]d\\Omega=\[0,1\]^\{d\},d\>2d\>2, andℓ​\(y,z\)=‖y−z‖22\\ell\(y,z\)=\\\|y\-z\\\|\_\{2\}^\{2\}denotes the squared Euclidean distance\. LetY,Z1,…​ZnY,Z\_\{1\},\\ldots Z\_\{n\}denote i\.i\.d\. random variables following a distributionℙ\\mathbb\{P\}with densitypp,𝔻=\{Z1,…,Zn\}\\mathbb\{D\}=\\\{Z\_\{1\},\\ldots,Z\_\{n\}\\\}denote the resulting i\.i\.d\. dataset, andℛ​\(𝔻\)=𝔼Y​\[mink∈\[\[1,n\]\]⁡‖Y−Zk‖22\]\\mathcal\{R\}\(\\mathbb\{D\}\)=\\mathbb\{E\}\_\{Y\}\\left\[\\;\\min\_\{k\\in\[\\\!\[1,n\]\\\!\]\}\\\|Y\-Z\_\{k\}\\\|\_\{2\}^\{2\}\\;\\right\]denote the resulting quantization error\. Then

𝔼𝔻​ℛ​\(𝔻\)​∼n→∞​Jd​∫\[0,1\]dp​\(y\)\(d−2\)/d​dy​1n2/d,\\mathbb\{E\}\_\{\\mathbb\{D\}\}\\mathcal\{R\}\(\\mathbb\{D\}\)\\underset\{n\\to\\infty\}\{\\sim\}J\_\{d\}\\int\_\{\[0,1\]^\{d\}\}p\(y\)^\{\(d\-2\)/d\}\\mathrm\{d\}y\\;\\frac\{1\}\{n^\{2/d\}\}\\;,\(23\)where

Jd=1π​Γ​\(2d\+1\)​Γ​\(d2\+1\)2/dJ\_\{d\}=\\frac\{1\}\{\\pi\}\\Gamma\\left\(\\frac\{2\}\{d\}\+1\\right\)\\Gamma\\left\(\\frac\{d\}\{2\}\+1\\right\)^\{2/d\}\(24\)

For an arbitrary manifoldΩ\\Omega, we define the density\-dependent volume functional appearing in Eq\.[23](https://arxiv.org/html/2605.21692#A2.E23)by

𝒱d​\(p\)=∫Ωp​\(x\)\(d−2\)/d​dx\.\\mathcal\{V\}\_\{d\}\(p\)=\\int\_\{\\Omega\}p\(x\)^\{\(d\-2\)/d\}\\mathrm\{d\}x\\;\.\(25\)
Interestingly, both optimal quantization and random nearest\-neighbor approximation exhibit the same asymptotic scaling law inn−2/dΩn^\{\-2/d\_\{\\Omega\}\}\. In both cases, the asymptotic behavior decomposes into: \(i\) a universal geometric constant, independent of the data distribution \(JdJ\_\{d\}, Eq\.[24](https://arxiv.org/html/2605.21692#A2.E24), orJd∗J\_\{d\}^\{\*\}, Eq\.[20](https://arxiv.org/html/2605.21692#A2.E20)\), and \(ii\) a density\-dependent volume functional \(𝒱d\\mathcal\{V\}\_\{d\}, Eq\.[25](https://arxiv.org/html/2605.21692#A2.E25), or𝒱d∗\\mathcal\{V\}\_\{d\}^\{\*\}, Eq\.[22](https://arxiv.org/html/2605.21692#A2.E22)\)\. Furthermore, both volume functionals reduce to the same geometric scaling𝒱d​\(p\)=𝒱d∗​\(p\)=\|Ω\|2/d\\mathcal\{V\}\_\{d\}\(p\)=\\mathcal\{V\}\_\{d\}^\{\*\}\(p\)=\|\\Omega\|^\{2/d\}for a uniform densitypp\. Our work unifies these two asymptotic regimes \(Theorems[7](https://arxiv.org/html/2605.21692#Thmtheorem7)and[8](https://arxiv.org/html/2605.21692#Thmtheorem8)\) under a common geometric framework\.

Despite these similarities, Theorem[8](https://arxiv.org/html/2605.21692#Thmtheorem8)and Zador’s theorem describe fundamentally different types of asymptotic results\. Theorem[8](https://arxiv.org/html/2605.21692#Thmtheorem8)characterizes the quantization error averaged over alli\.i\.d\.datasets of sizenn, whereas Zador’s theorem characterizes the quantization error of a specific \(optimal\) point configuration\. From a learning perspective, results in expectation are insufficient to characterize the behavior of a model trained on a specific dataset\. One of the main contributions of our work is to bridge these two asymptotic regimes by extending Theorem[8](https://arxiv.org/html/2605.21692#Thmtheorem8)to a convergence result in probability\.

## Appendix CNon\-conditional tasks

### C\.1Memorizing networks and representation gap

Let us first consider the case of a non\-conditional prediction task\. This setting corresponds to unconditional generative modeling, where the goal is to learn a probability distribution supported onΩ⊂𝒴\\Omega\\subset\\mathcal\{Y\}that captures the geometric structure of the data manifold\.

Popular approaches for generative modeling include diffusion models\(Hoet al\.,[2020](https://arxiv.org/html/2605.21692#bib.bib141); Songet al\.,[2022](https://arxiv.org/html/2605.21692#bib.bib159)\), Variational Auto Encoders \(VAE\)\(Kingma and Welling,[2022](https://arxiv.org/html/2605.21692#bib.bib167)\), Generative Adversarial Networks \(GAN\)\(Goodfellowet al\.,[2014](https://arxiv.org/html/2605.21692#bib.bib168)\)or normalizing flows\(Rezende and Mohamed,[2016](https://arxiv.org/html/2605.21692#bib.bib169)\)\. Among them, diffusion models can be shown to converge toward the empirical distribution1\|𝔻\|​∑y∈𝔻δy\\frac\{1\}\{\|\\mathbb\{D\}\|\}\\sum\_\{y\\in\\mathbb\{D\}\}\\delta\_\{y\}when they minimize their training objective\(Song and Ermon,[2019](https://arxiv.org/html/2605.21692#bib.bib136)\)\.

We will focus on this class of models hereafter\. In this case, the empirical distribution corresponds to the prediction spaceΩf\\Omega\_\{f\}learned by the modelff, which can be seen as a discrete approximation ofΩ\\Omega\. We can compare this discrete prediction spaceΩf\\Omega\_\{f\}toΩ\\Omegausing the quantization error\(Zador,[1982](https://arxiv.org/html/2605.21692#bib.bib179)\)\. This metric can be extended in the more general case whereΩf\\Omega\_\{f\}may be continuous\. We will refer to this distance as therepresentation gap\.

###### Definition 3\(Representation gap\)\.

LetΩ\\Omegadenote the data manifold andΩf\\Omega\_\{f\}denote the model’s prediction space\. We define the representation gap as follows:

ℛ​\(Ω,Ωf\)=∫Ωinfz∈Ωfℓ​\(y,z\)​p​\(y\)​d​y\.\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)=\\int\_\{\\Omega\}\\inf\_\{z\\in\\Omega\_\{f\}\}\\ell\(y,z\)\\;p\(y\)\\;\\mathrm\{d\}y\\;\.\(26\)

The representation gapℛ​\(Ω,Ωf\)\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)depends on how the dataset𝔻\\mathbb\{D\}is sampled fromΩ\\Omega\. It is typical to assume that𝔻\\mathbb\{D\}is sampledi\.i\.d\.from the data distributionpp, and we denoterandom representation gapthe corresponding quantity\.

###### Definition 4\(Random representation gap\)\.

LetΩ\\Omegadenote the data manifold,𝔻\\mathbb\{D\}a dataset of sizennsampled i\.i\.d\. from p, andΩf=Ωf​\(𝔻\)\\Omega\_\{f\}=\\Omega\_\{f\}\(\\mathbb\{D\}\)denote the prediction space of a modelfftrained on𝔻\\mathbb\{D\}\. We define the random representation gap as the random variable

ℛn=∫Ωinfz∈Ωfℓ​\(y,z\)​p​\(y\)​d​y\.\\mathcal\{R\}\_\{n\}=\\int\_\{\\Omega\}\\inf\_\{z\\in\\Omega\_\{f\}\}\\ell\(y,z\)\\;p\(y\)\\;\\mathrm\{d\}y\\;\.

Although theoretical analyses of generalization typically assume that training and test data arei\.i\.d\.\(Shalev\-Shwartz and Ben\-David,[2014](https://arxiv.org/html/2605.21692#bib.bib204)\), the validity of this hypothesis has been questioned in the literature\(Mohri and Rostamizadeh,[2008](https://arxiv.org/html/2605.21692#bib.bib156)\)\. Indeed, datasets are often collected to cover the diversity of the task, modulo its known invariants\(Torralba and Efros,[2011](https://arxiv.org/html/2605.21692#bib.bib208)\)\. Motivated by this observation, we also consider the setting where𝔻\\mathbb\{D\}is optimally diverse, i\.e\. minimizes the representation gap, and we denoteoptimal representation gapthe corresponding quantity\.

###### Definition 5\(Optimal representation gap\)\.

LetΩ\\Omegadenote the data manifold\. For each dataset𝔻⊂Ω\\mathbb\{D\}\\subset\\Omega, we denote byΩf=Ωf​\(𝔻\)\\Omega\_\{f\}=\\Omega\_\{f\}\(\\mathbb\{D\}\)the prediction space of a modelfftrained on𝔻\\mathbb\{D\}\. We define the optimal representation gap as

ℛn∗=inf𝔻⊂Ω,\|𝔻\|=nℛ​\(Ω,Ωf​\(𝔻\)\)\.\\mathcal\{R\}\_\{n\}^\{\*\}=\\inf\_\{\\mathbb\{D\}\\subset\\Omega,\\;\|\\mathbb\{D\}\|=n\}\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\(\\mathbb\{D\}\)\)\.

Note that the random representation gapℛn\\mathcal\{R\}\_\{n\}is a random variable while the optimal representation gapℛn∗\\mathcal\{R\}\_\{n\}^\{\*\}is a scalar value\. Both quantities are notoriously difficult to study, even in the discrete case\(Graf and Luschgy,[2007](https://arxiv.org/html/2605.21692#bib.bib182)\)\. However, they become amenable to analysis in the asymptotic regime\.

### C\.2Asymptotic representation gap in the Euclidean setting

The asymptotic scaling of the representation gap naturally characterizes how efficiently the geometry of a task can be learned from a finite dataset\.

###### Proposition 1\(Optimal representation gap\)\.

Let us assume thatΩ\\Omegais Lebesgue\-measurable with positive measure\. Then, the optimal representation gap of a diffusion modelffminimizing its training objective[3](https://arxiv.org/html/2605.21692#S4.E3)on a training dataset of sizennis

ℛn∗​∼n→\+∞​Jd∗​𝒱d∗​\(p\)n2/d\.\\mathcal\{R\}\_\{n\}^\{\*\}\\underset\{n\\rightarrow\+\\infty\}\{\\scalebox\{1\.5\}\{$\\sim$\}\}\\frac\{J\_\{d\}^\{\*\}\\mathcal\{V\}\_\{d\}^\{\*\}\(p\)\}\{n^\{2/d\}\}\\;\.\(27\)

###### Proof\.

This is a corollary of Zador Theorem[6](https://arxiv.org/html/2605.21692#Thmtheorem6)\. ∎

This result is remarkable, since it provides an asymptotic equivalent of the representation gap as the dataset sizenngrows to infinity\. Most notably, the leading constant depends on the geometry ofΩ\\Omegaonly via a volume term𝒱d∗​\(p\)\\mathcal\{V\}\_\{d\}^\{\*\}\(p\)\.

### C\.3Asymptotic representation gap under the manifold hypothesis

It is possible to extend this result whenΩ\\Omegais a low\-dimensional manifold of the target space𝒴\\mathcal\{Y\}\. This setting is interesting because it captures the structure of the observation manifoldΩ\\Omega: even though the observation could a priori be an arbitrary point of𝒴\\mathcal\{Y\}, it is in effect restricted to the subspaceΩ\\Omega\.

###### Proposition 2\(Optimal representation gap under the manifold hypothesis\)\.

Assume thatΩ\\Omegais a compact RiemanniandΩd\_\{\\Omega\}\-manifold\. Then the optimal representation gap of a diffusion modelffminimizing its training objective[3](https://arxiv.org/html/2605.21692#S4.E3)on a training dataset𝔻\\mathbb\{D\}of sizennsatisfies

ℛn∗​∼n→\+∞​JdΩ∗​𝒱dΩ∗​\(p\)n2/dΩ\.\\mathcal\{R\}\_\{n\}^\{\*\}\\underset\{n\\rightarrow\+\\infty\}\{\\scalebox\{1\.5\}\{$\\sim$\}\}\\frac\{J\_\{d\_\{\\Omega\}\}^\{\*\}\\;\\mathcal\{V\}\_\{d\_\{\\Omega\}\}^\{\*\}\(p\)\}\{n^\{2/d\_\{\\Omega\}\}\}\\;\.\(28\)

###### Proof\.

This is a corollary of Theorem 2 inGruber \([2001](https://arxiv.org/html/2605.21692#bib.bib92)\)\(see also Theorem[7](https://arxiv.org/html/2605.21692#Thmtheorem7)\)\. The assumptions of the theorem are satisfied, sinceΩ\\Omegais compact and the squared geodesic distance satisfies the required growth condition\. We denote byJd∗J\_\{d\}^\{\*\}the corresponding asymptotic constant, consistently with the Euclidean formulation of Zador’s theorem\. ∎

This asymptotic evolution is similar to the general case described in Proposition[1](https://arxiv.org/html/2605.21692#Thmprop1), but leverages the structure ofΩ\\Omegavia the lower dimensiondΩd\_\{\\Omega\}\. Note that we recover Proposition[1](https://arxiv.org/html/2605.21692#Thmprop1)whenΩ\\Omegahas positive measure in𝒴\\mathcal\{Y\}\. Again, we highlight that the leading constant depends on the geometry ofΩ\\Omegaonly via a volume term𝒱dΩ∗​\(p\)\\mathcal\{V\}\_\{d\_\{\\Omega\}\}^\{\*\}\(p\)\. Moreover, it can be proved that the optimal data placement for𝔻\\mathbb\{D\}is uniformly distributed inΩ\\Omegawhenppis uniform \(cf\. point 2\.82 inGruber \([2001](https://arxiv.org/html/2605.21692#bib.bib92)\)\)\.

Leveraging results from point process theory, we describe the asymptotic random representation gap in the following Proposition\.

###### Proposition 3\(Random representation gap under the manifold hypothesis\)\.

Assume thatΩ\\Omegais a compactdΩd\_\{\\Omega\}\-dimensional Riemannian manifold without boundary andppis continuous and strictly positive onΩ\\Omega\. Then the random representation gap of a diffusion modelffminimizing its training objective[3](https://arxiv.org/html/2605.21692#S4.E3)on a training dataset𝔻\\mathbb\{D\}of sizennsatisfies

ℛn∼ℙJdΩ​𝒱dΩ​\(p\)n2/dΩ,\\mathcal\{R\}\_\{n\}\\sim\_\{\\mathbb\{P\}\}\\frac\{J\_\{d\_\{\\Omega\}\}\\mathcal\{V\}\_\{d\_\{\\Omega\}\}\(p\)\}\{n^\{2/d\_\{\\Omega\}\}\}\\;,\(29\)where we have definedJd=1π​Γ​\(1\+d2\)2/d​Γ​\(1\+2d\)J\_\{d\}=\\frac\{1\}\{\\pi\}\\Gamma\(1\+\\frac\{d\}\{2\}\)^\{2/d\}\\Gamma\(1\+\\frac\{2\}\{d\}\)and𝒱d​\(p\)=∫Ωp​\(x\)\(d−2\)/d​dx\\mathcal\{V\}\_\{d\}\(p\)=\\int\_\{\\Omega\}p\(x\)^\{\(d\-2\)/d\}\\mathrm\{d\}x\.

###### Proof\.

The idea is to extend the proof of Theorem 2\.3 inBiau and Devroye \([2015](https://arxiv.org/html/2605.21692#bib.bib7)\)to the manifold setting\. Intuitively, the argument relies on the following three observations: \(i\) the random representation gapℛn\\mathcal\{R\}\_\{n\}can be written as the expectation overYYof the random variableAn​\(Y\)=n2/d​minz∈𝔻⁡ℓ​\(Y,z\)A\_\{n\}\(Y\)=n^\{2/d\}\\min\_\{z\\in\\mathbb\{D\}\}\\ell\(Y,z\), up to normalization factors; \(ii\) for fixedy∈Ωy\\in\\Omega, the asymptotic behavior ofAn​\(y\)A\_\{n\}\(y\)is entirely characterized by its tail distributionℙ​\(An​\(y\)\>t\)\\mathbb\{P\}\(A\_\{n\}\(y\)\>t\); \(iii\) under the manifold hypothesis, one can show thatℙ​\(An​\(y\)\>t\)→exp⁡\(−p​\(y\)​Vd​td/2\)\\mathbb\{P\}\(A\_\{n\}\(y\)\>t\)\\rightarrow\\exp\(\-p\(y\)V\_\{d\}t^\{d/2\}\), which gives the result after integration overttandyy\.

We now prove this result formally\. We writed=dΩd=d\_\{\\Omega\}for simplicity\. Letz1,…,zn∼pz\_\{1\},\\ldots,z\_\{n\}\\sim p,𝔻=\{z1,…,zn\}\\mathbb\{D\}=\\\{z\_\{1\},\\ldots,z\_\{n\}\\\}, and defineAn​\(y\)=n2/d​min1≤j≤n⁡ℓ​\(y,zj\)A\_\{n\}\(y\)=n^\{2/d\}\\min\_\{1\\leq j\\leq n\}\\ell\(y,z\_\{j\}\)\. LetA​\(y\)A\(y\)denote a random variable such thatℙ​\(A​\(y\)\>t\)=exp⁡\(−p​\(y\)​Vd​td/2\)\\mathbb\{P\}\(A\(y\)\>t\)=\\exp\(\-p\(y\)V\_\{d\}t^\{d/2\}\), whereVdV\_\{d\}denotes the volume of the unit\-ball inℝd\\mathbb\{R\}^\{d\}\.

We know thatℙ​\(An​\(y\)\>t\)=\(1−ℙ​\(B​\(y,t​n−1/d\)\)\)n\\mathbb\{P\}\(A\_\{n\}\(y\)\>t\)=\\left\(1\-\\mathbb\{P\}\\left\(B\\left\(y,\\sqrt\{t\}\\;n^\{\-1/d\}\\right\)\\right\)\\right\)^\{n\}, whereB​\(y,t​n−1/d\)B\\left\(y,\\sqrt\{t\}\\;n^\{\-1/d\}\\right\)denote the ball of radiust​n−1/d\\sqrt\{t\}\\;n^\{\-1/d\}centered onyyin the manifoldΩ\\Omega\. Moreover, we know that the volume of a ballB​\(y,ε\)B\(y,\\varepsilon\)in a manifoldΩ\\Omegaof dimensionddcan be approximated by\|B​\(y,ε\)\|=Vd​εd\+o​\(εd\)\|B\(y,\\varepsilon\)\|=V\_\{d\}\\varepsilon^\{d\}\+o\(\\varepsilon^\{d\}\)\(see for instance Eq\. 1 inBarilariet al\.\([2018](https://arxiv.org/html/2605.21692#bib.bib199)\)\)\. By continuity ofpp, we then haveℙ​\(B​\(y,ε\)\)=p​\(y\)​Vd​εd\+o​\(εd\)\\mathbb\{P\}\(B\(y,\\varepsilon\)\)=p\(y\)V\_\{d\}\\varepsilon^\{d\}\+o\(\\varepsilon^\{d\}\)\. Therefore, we deduce

ℙ​\(An​\(y\)\>t\)=\(1−ℙ​\(B​\(y,tn1/d\)\)\)n​→n→∞​exp⁡\(−p​\(y\)​Vd​td/2\)=ℙ​\(A​\(y\)\>t\)\.\\mathbb\{P\}\(A\_\{n\}\(y\)\>t\)=\\left\(1\-\\mathbb\{P\}\\left\(B\\left\(y,\\frac\{\\sqrt\{t\}\}\{n^\{1/d\}\}\\right\)\\right\)\\right\)^\{n\}\\underset\{n\\rightarrow\\infty\}\{\\rightarrow\}\\;\\exp\(\-p\(y\)V\_\{d\}t^\{d/2\}\)=\\mathbb\{P\}\(A\(y\)\>t\)\\;\.In particular,An​\(y\)→dA​\(y\)A\_\{n\}\(y\)\\rightarrow\_\{d\}A\(y\)in distribution\. Moreover, sinceℙ​\(B​\(y,r\)\)=p​\(y\)​Vd​rd\+o​\(rd\)\\mathbb\{P\}\(B\(y,r\)\)=p\(y\)V\_\{d\}\\;r^\{d\}\+o\(r^\{d\}\), there existsc​\(y\)\>0c\(y\)\>0such that for sufficiently largenn,ℙ​\(An​\(y\)\>t\)≤exp⁡\(−c​\(y\)​td/2\)\\mathbb\{P\}\(A\_\{n\}\(y\)\>t\)\\leq\\exp\(\-c\(y\)t^\{d/2\}\)\. Therefore,\(An​\(y\)\)n\(A\_\{n\}\(y\)\)\_\{n\}is uniformly integrable\.

We deduceAn​\(y\)→L1A​\(y\)A\_\{n\}\(y\)\\rightarrow\_\{L^\{1\}\}A\(y\)\(Theorem 4\.6\.3 inDurrett \([2019](https://arxiv.org/html/2605.21692#bib.bib11)\)\), and then𝔼Y∼p​\[An​\(Y\)\|𝔻\]→ℙ𝔼Y∼p​A​\(Y\)\\mathbb\{E\}\_\{Y\\sim p\}\[A\_\{n\}\(Y\)\|\\mathbb\{D\}\]\\rightarrow\_\{\\mathbb\{P\}\}\\mathbb\{E\}\_\{Y\\sim p\}A\(Y\)\(Example 4\.6\.11 inDurrett \([2019](https://arxiv.org/html/2605.21692#bib.bib11)\)\)\. We conclude by observingn2/d​ℛ​\(Ω,𝔻\)=𝔼Y∼p​\[An​\(Y\)\|𝔻\]n^\{2/d\}\\mathcal\{R\}\(\\Omega,\\mathbb\{D\}\)=\\mathbb\{E\}\_\{Y\\sim p\}\[A\_\{n\}\(Y\)\|\\mathbb\{D\}\]on one hand\. On the other hand,

𝔼Y∼p​A​\(Y\)=∫0∞∫Ωexp⁡\(−p​\(y\)​Vd​td/2\)​p​\(y\)​dy​dt=Jd​𝒱d​\(p\),\\mathbb\{E\}\_\{Y\\sim p\}A\(Y\)=\\int\_\{0\}^\{\\infty\}\\int\_\{\\Omega\}\\exp\(\-p\(y\)V\_\{d\}\\;t^\{d/2\}\)p\(y\)\\mathrm\{d\}y\\mathrm\{d\}t=J\_\{d\}\\mathcal\{V\}\_\{d\}\(p\)\\;,\(by change of variableu=p​\(y\)​Vd​td/2u=p\(y\)V\_\{d\}t^\{d/2\}and using the properties of theΓ\\Gammaintegral\), so that

ℛ​\(Ω,𝔻\)=Jd​𝒱d​\(p\)​n−2/d\+oℙ​\(n−2/d\)\.\\mathcal\{R\}\(\\Omega,\\mathbb\{D\}\)=J\_\{d\}\\mathcal\{V\}\_\{d\}\(p\)n^\{\-2/d\}\+o\_\{\\mathbb\{P\}\}\(n^\{\-2/d\}\)\\;\.∎

Unlike classical nearest\-neighbor asymptotics, which characterize the quantization error only in expectation over random datasets, Proposition[3](https://arxiv.org/html/2605.21692#Thmprop3)establishes convergence in probability for individual datasets\. This distinction is important from a learning perspective, since neural networks are trained on a specific realized dataset rather than on an average over datasets\. As such, our result is more directly relevant to the practice of neural network training\.

### C\.4Asymptotic representation gap for equivariant models

In practice,ℱΘ\\mathcal\{F\}\_\{\\Theta\}has limited expressivity, which introduces biases in the minimizerf=argminθ∈Θ​ℒ​\(θ\)f=\\mathrm\{argmin\}\_\{\\theta\\in\\Theta\}\\mathcal\{L\}\(\\theta\)\. Typically, the architecture of the neural network is chosen so thatfθf\_\{\\theta\}respects the symmetries ofΩ\\Omega, and has therefore higher generalization capabilities\. Remarkably, the authors ofKamb and Ganguli \([2025](https://arxiv.org/html/2605.21692#bib.bib88)\)have shown in the context of diffusion models that these architectural constraints virtually augment the diversity of the dataset𝔻\\mathbb\{D\}via the symmetry group introduced by the architecture\.

The following result is an extension of Theorem B\.3 inKamb and Ganguli \([2025](https://arxiv.org/html/2605.21692#bib.bib88)\)to general symmetry groupsGG\. More precisely, we will focus our attention on Lie groups, which naturally describe many symmetries appearing in neural networks\(Bronsteinet al\.,[2021](https://arxiv.org/html/2605.21692#bib.bib170)\)\. They are also used in various fields such as physics, where they reflect the structure and symmetries of many physical systems\(Gilmore,[2006](https://arxiv.org/html/2605.21692#bib.bib171); Georgi,[2000](https://arxiv.org/html/2605.21692#bib.bib172)\)\. This makes them particularly relevant for our purposes\.

###### Proposition 4\(Virtual augmentation of a dataset by a symmetry group\)\.

Let us make the following assumptions

\(i\)ffis a trained diffusion model equivariant toGG\.

\(ii\) G is a Lie group acting smoothly on the Riemannian manifoldΩ\\Omega\.

\(iii\) The minimumminz∈G​\(𝔻\)⁡ℓ​\(y,z\)\\min\_\{z\\in G\(\\mathbb\{D\}\)\}\\ell\(y,z\)is reached at a unique pointy∗=argminz∈G​\(𝔻\)​ℓ​\(y,z\)y^\{\*\}=\\mathrm\{argmin\}\_\{z\\in G\(\\mathbb\{D\}\)\}\\ell\(y,z\)for ally∈𝒴y\\in\\mathcal\{Y\}\.

\(iv\) Letyty\_\{t\}denote the denoising trajectory from the Gaussian distributionπT\\pi\_\{T\}, standard reverse diffusion process∂tyt=−γt​\(yt\+f​\(yt,t\)\)\\partial\_\{t\}y\_\{t\}=\-\\gamma\_\{t\}\(y\_\{t\}\+f\(y\_\{t\},t\)\)\. Assume thatyty\_\{t\}converges and∂tyt\\partial\_\{t\}y\_\{t\}is bounded for each initial pointyTy\_\{T\}\.

Then, the denoising trajectory ends aty0∈G​\(𝔻\)y\_\{0\}\\in G\(\\mathbb\{D\}\)\.

If we further assume each dataset pointz∈𝔻z\\in\\mathbb\{D\}is a fixed point of thef​\(⋅,t\)f\(\\cdot,t\)for alltt, then each pointz∈G​\(𝔻\)z\\in G\(\\mathbb\{D\}\)is a limit point of the reverse diffusion process\.

Proposition[4](https://arxiv.org/html/2605.21692#Thmprop4)essentially states that under mild assumptions, an equivariant diffusion modelffwill generate samples in the virtually augmented datasetG​\(𝔻\)G\(\\mathbb\{D\}\)\. This is because the vision of the modelffis blurred due to its equivariance toGG, so that it cannot distinguish points along the orbitsG​\(y\)G\(y\)of the dataset pointsy∈𝔻y\\in\\mathbb\{D\}\.

The hypothesis\(i\)\(i\)states that the modelffis a global minimum of its training objectiveℒ\\mathcal\{L\}\. The hypothesis\(i​i\)\(ii\)restricts our attention to Lie groupsGG, as discussed above\. The point\(i​i​i\)\(iii\)avoids the degenerate case where the initial pointyyis equidistant to a subset of the orbit of the datasetG​\(𝔻\)G\(\\mathbb\{D\}\)\. The point\(i​v\)\(iv\)is a slightly relaxed form of a technical assumption introduced in Theorem B\.3 ofKamb and Ganguli \([2025](https://arxiv.org/html/2605.21692#bib.bib88)\)\. Finally, the fixed\-point hypothesis captures the fact that each pointz∈𝔻z\\in\\mathbb\{D\}is a local attractor of the score function, since the empirical distribution is discrete in our setting\.

The proof of Proposition[4](https://arxiv.org/html/2605.21692#Thmprop4)relies on the following observation: the score function can be written as an integral over the orbitsG​\(z\)G\(z\)of each data pointz∈𝔻z\\in\\mathbb\{D\}, where each pointzzis weighted by the distribution

Wt​\(z\)=𝒩​\(y\|αt​z,\(1−αt\)​I\)∫G​\(𝔻\)𝒩​\(y\|αt​z′,\(1−αt\)​I\)​dz′\.W\_\{t\}\(z\)=\\frac\{\\mathcal\{N\}\\left\(y\|\\sqrt\{\\alpha\_\{t\}\}z,\(1\-\\alpha\_\{t\}\)I\\right\)\}\{\\int\_\{G\(\\mathbb\{D\}\)\}\\mathcal\{N\}\\left\(y\|\\sqrt\{\\alpha\_\{t\}\}z^\{\\prime\},\(1\-\\alpha\_\{t\}\)I\\right\)\\mathrm\{d\}z^\{\\prime\}\}\\;\.\(30\)In the case where the groupGGis finite, we can see thatWt​\(z\)W\_\{t\}\(z\)acts as a softmax that peaks whenz∗z^\{\*\}ast→0t\\rightarrow 0\. In the more general case whereGGis not finite, we can use a Laplace approximation to show thatWt​\(z\)W\_\{t\}\(z\)concentrates the probability mass around the minimizerz∗z^\{\*\}whent→0t\\rightarrow 0\. Therefore, the denoising trajectory is attracted toward the orbitG​\(𝔻\)G\(\\mathbb\{D\}\)\.

###### Lemma 1\(Laplace approximation\)\.

Let G denote a Lie group acting smoothly onΩ\\Omega,αt\\alpha\_\{t\}a continuous positive noise schedule satisfyingαt→t→01\\alpha\_\{t\}\\rightarrow\_\{t\\rightarrow 0\}1,y∈𝒴y\\in\\mathcal\{Y\}an arbitrary point,ddthe dimension ofG​\(𝔻\)G\(\\mathbb\{D\}\), andhha bounded continuous non\-negative function onG​\(𝔻\)G\(\\mathbb\{D\}\)\. Assume thatyyhas a unique closest pointy∗∈G​\(𝔻\)̊y^\{\*\}\\in\\mathring\{G\(\\mathbb\{D\}\)\}, the interior of the orbit\. Defineβt=2​1−αtαt\\beta\_\{t\}=2\\frac\{1\-\\alpha\_\{t\}\}\{\\alpha\_\{t\}\}a temperature scaling\. Then, we have

∫G​\(𝔻\)h​\(z\)​𝒩​\(y\|αt​z,\(1−αt\)​I\)​dz\\displaystyle\\int\_\{G\(\\mathbb\{D\}\)\}h\(z\)\\mathcal\{N\}\\left\(y\|\\sqrt\{\\alpha\_\{t\}\}z,\(1\-\\alpha\_\{t\}\)I\\right\)\\mathrm\{d\}z=t→0​h​\(y∗\)​e−‖y∗−y‖2/βt​\(2​π​βt\)d/2\\displaystyle\\underset\{t\\rightarrow 0\}\{=\}h\(y^\{\*\}\)\\;e^\{\-\\\|y^\{\*\}\-y\\\|^\{2\}/\\beta\_\{t\}\}\\;\(2\\pi\\beta\_\{t\}\)^\{d/2\}\(31\)\+o​\(e−‖y∗−y‖2/βt​βtd/2\)\.\\displaystyle\+o\\left\(\\;e^\{\-\\\|y^\{\*\}\-y\\\|^\{2\}/\\beta\_\{t\}\}\\;\\beta\_\{t\}^\{d/2\}\\right\)\\;\.

###### Proof\.

Let us denote byI​\(t\)=∫G​\(𝔻\)h​\(z\)​𝒩​\(y\|αt​z,\(1−αt\)​I\)​dzI\(t\)=\\int\_\{G\(\\mathbb\{D\}\)\}h\(z\)\\mathcal\{N\}\\left\(y\|\\sqrt\{\\alpha\_\{t\}\}z,\(1\-\\alpha\_\{t\}\)I\\right\)\\mathrm\{d\}zthe left term in Eq\.[31](https://arxiv.org/html/2605.21692#A3.E31)\. Informally, the proof of Lemma[1](https://arxiv.org/html/2605.21692#Thmlemma1)then relies on the two following approximations:

I​\(t\)=∫G​\(𝔻\)h​\(z\)​e−‖z−yαt‖2/βt​dz≈∫G​\(𝔻\)h​\(z\)​e−‖z−y‖2/βt​dz≈h​\(y∗\)​e−‖y∗−y‖2/βt​\(2​π​βt\)d/2\.I\(t\)=\\int\_\{G\(\\mathbb\{D\}\)\}h\(z\)e^\{\-\\\|z\-\\frac\{y\}\{\\alpha\_\{t\}\}\\\|^\{2\}/\\beta\_\{t\}\}\\mathrm\{d\}z\\ \\approx\\int\_\{G\(\\mathbb\{D\}\)\}h\(z\)e^\{\-\\\|z\-y\\\|^\{2\}/\\beta\_\{t\}\}\\mathrm\{d\}z\\approx h\(y^\{\*\}\)e^\{\-\\\|y^\{\*\}\-y\\\|^\{2\}/\\beta\_\{t\}\}\(2\\pi\\beta\_\{t\}\)^\{d/2\}\\;\.The first approximation comes from integrating‖z−yαt‖2=‖z−y‖2\+O​\(βt\)\\\|z\-\\frac\{y\}\{\\alpha\_\{t\}\}\\\|^\{2\}=\\\|z\-y\\\|^\{2\}\+O\(\\beta\_\{t\}\)over the orbitG​\(𝔻\)G\(\\mathbb\{D\}\), and the second approximation is an extension of Laplace approximation on measurable subsets ofℝd\\mathbb\{R\}^\{d\}\. It expresses that the Gaussian kernele−‖z−y‖2/βte^\{\-\\\|z\-y\\\|^\{2\}/\\beta\_\{t\}\}concentrates mass at the minimizery∗y^\{\*\}, with a curvature term\(2​π​βt\)d/2\(2\\pi\\beta\_\{t\}\)^\{d/2\}\.

Let us now prove these two approximations\. First observe that

‖z−yαt‖2−‖y∗−yαt‖2=‖z−y‖2−‖y∗−y‖2\+2​αt−1αt​⟨z−y∗\|y⟩,\\\|z\-\\frac\{y\}\{\\alpha\_\{t\}\}\\\|^\{2\}\-\\\|y^\{\*\}\-\\frac\{y\}\{\\alpha\_\{t\}\}\\\|^\{2\}=\\\|z\-y\\\|^\{2\}\-\\\|y^\{\*\}\-y\\\|^\{2\}\+2\\frac\{\\sqrt\{\\alpha\_\{t\}\}\-1\}\{\\sqrt\{\\alpha\_\{t\}\}\}\\langle z\-y^\{\*\}\|y\\rangle,so that by exponentiation and integration, we have

∫G​\(𝔻\)h​\(z\)​e−‖z−yαt‖2/βt​dz=e−‖y∗−yαt‖2/βt​∫G​\(𝔻\)h​\(z\)​eαt2​\(1\+αt\)​⟨y∗−z\|y⟩​e\(‖y∗−y‖2−‖z−y‖2\)/βt​dz⏟J​\(t\)\.\\int\_\{G\(\\mathbb\{D\}\)\}h\(z\)e^\{\-\\\|z\-\\frac\{y\}\{\\alpha\_\{t\}\}\\\|^\{2\}/\\beta\_\{t\}\}\\mathrm\{d\}z=e^\{\-\\\|y^\{\*\}\-\\frac\{y\}\{\\alpha\_\{t\}\}\\\|^\{2\}/\\beta\_\{t\}\}\\underbrace\{\\int\_\{G\(\\mathbb\{D\}\)\}h\(z\)e^\{\\frac\{\\sqrt\{\\alpha\_\{t\}\}\}\{2\(1\+\\sqrt\{\\alpha\_\{t\}\}\)\}\\langle y^\{\*\}\-z\|y\\rangle\}\\;e^\{\(\\\|y^\{\*\}\-y\\\|^\{2\}\-\\\|z\-y\\\|^\{2\}\)/\\beta\_\{t\}\}\\mathrm\{d\}z\}\_\{J\(t\)\}\.
The noise scheduleαt\\alpha\_\{t\}is bounded in\[0,1\]\[0,1\], so thate−\|⟨y∗−z\|y⟩\|≤eαt2​\(1\+αt\)​⟨y∗−z\|y⟩≤e\|⟨y∗−z\|y⟩\|e^\{\-\|\\langle y^\{\*\}\-z\|y\\rangle\|\}\\leq e^\{\\frac\{\\sqrt\{\\alpha\_\{t\}\}\}\{2\(1\+\\sqrt\{\\alpha\_\{t\}\}\)\}\\langle y^\{\*\}\-z\|y\\rangle\}\\leq e^\{\|\\langle y^\{\*\}\-z\|y\\rangle\|\}\. Let us define

J−​\(t\)=∫G​\(𝔻\)h​\(z\)​e−\|⟨y∗−z\|y⟩\|​e\(‖y∗−y‖2−‖z−y‖2\)/βt​dz,J\_\{\-\}\(t\)=\\int\_\{G\(\\mathbb\{D\}\)\}h\(z\)\\;e^\{\-\|\\langle y^\{\*\}\-z\|y\\rangle\|\}\\;e^\{\(\\\|y^\{\*\}\-y\\\|^\{2\}\-\\\|z\-y\\\|^\{2\}\)/\\beta\_\{t\}\}\\mathrm\{d\}z\\;,a lower bound ofJ​\(t\)J\(t\)\.

Then we can apply Corollary 3\.4 inKirwin \([2010](https://arxiv.org/html/2605.21692#bib.bib95)\)toJ​\(t\)J\(t\)in order to obtain thatJ−​\(t\)​=t→0​h​\(y∗\)​\(2​π​βt\)d/2\+o​\(βtd/2\)J\_\{\-\}\(t\)\\underset\{t\\rightarrow 0\}\{=\}h\(y^\{\*\}\)\(2\\pi\\beta\_\{t\}\)^\{d/2\}\+o\(\\beta\_\{t\}^\{d/2\}\)\. Indeed, the conditions of this Corollary are met \(modulo a change of variable\), sinceG​\(𝔻\)G\(\\mathbb\{D\}\)is a measurable set which containsy∗y^\{\*\}as an interior point,z↦‖y∗−y‖2−‖z−y‖2z\\mapsto\\\|y^\{\*\}\-y\\\|^\{2\}\-\\\|z\-y\\\|^\{2\}is twice differentiable and attains its unique minimum value of 0 aty∗y^\{\*\},z↦h​\(z\)​e−\|⟨y∗−z\|y⟩\|z\\mapsto h\(z\)e^\{\-\|\\langle y^\{\*\}\-z\|y\\rangle\|\}is a continuous function onG​\(𝔻\)G\(\\mathbb\{D\}\)evaluating ath​\(y∗\)h\(y^\{\*\}\)ony∗y^\{\*\}, and1/βt​→t→0\+∞1/\\beta\_\{t\}\\underset\{t\\rightarrow 0\}\{\\rightarrow\}\+\\infty\.

Likewise, we can also prove that

J\+​\(t\)=∫G​\(𝔻\)h​\(z\)​e\|⟨y∗−z\|y⟩\|​e\(‖y∗−y‖2−‖z−y‖2\)/βt​dz​=t→0​h​\(y∗\)​\(2​π​βt\)d/2\+o​\(βtd/2\)\.J\_\{\+\}\(t\)=\\int\_\{G\(\\mathbb\{D\}\)\}h\(z\)\\;e^\{\|\\langle y^\{\*\}\-z\|y\\rangle\|\}\\;e^\{\(\\\|y^\{\*\}\-y\\\|^\{2\}\-\\\|z\-y\\\|^\{2\}\)/\\beta\_\{t\}\}\\mathrm\{d\}z\\underset\{t\\rightarrow 0\}\{=\}h\(y^\{\*\}\)\(2\\pi\\beta\_\{t\}\)^\{d/2\}\+o\(\\beta\_\{t\}^\{d/2\}\)\\;\.Therefore, we deduce by squeezing thatJ​\(t\)​=t→0​h​\(y∗\)​\(2​π​βt\)d/2\+o​\(βtd/2\)J\(t\)\\underset\{t\\rightarrow 0\}\{=\}h\(y^\{\*\}\)\(2\\pi\\beta\_\{t\}\)^\{d/2\}\+o\(\\beta\_\{t\}^\{d/2\}\), and we can conclude

I​\(t\)=e−‖y∗−yαt‖2/βt​J​\(t\)​=t→0​h​\(y∗\)​e−‖y∗−y‖2/βt​\(2​π​βt\)d/2\+o​\(e−‖y∗−y‖2/βt​βtd/2\)\.I\(t\)=e^\{\-\\\|y^\{\*\}\-\\frac\{y\}\{\\alpha\_\{t\}\}\\\|^\{2\}/\\beta\_\{t\}\}J\(t\)\\underset\{t\\rightarrow 0\}\{=\}\\;h\(y^\{\*\}\)\\;e^\{\-\\\|y^\{\*\}\-y\\\|^\{2\}/\\beta\_\{t\}\}\\;\(2\\pi\\beta\_\{t\}\)^\{d/2\}\+o\\left\(\\;e^\{\-\\\|y^\{\*\}\-y\\\|^\{2\}/\\beta\_\{t\}\}\\;\\beta\_\{t\}^\{d/2\}\\right\)\\;\.∎

We can now prove Proposition[4](https://arxiv.org/html/2605.21692#Thmprop4)\.

###### Proof of Proposition[4](https://arxiv.org/html/2605.21692#Thmprop4)\.

By theorem B\.3 inKamb and Ganguli \([2025](https://arxiv.org/html/2605.21692#bib.bib88)\), the score function by the modelffcan be written

f​\(yt,t\)=−11−αt​∫G​\(𝔻\)\(y−αt​z\)​𝒩​\(y\|αt​z,\(1−αt\)​I\)​dz∫G​\(𝔻\)𝒩​\(y\|αt​z,\(1−αt\)​I\)​dz=11−αt​\(yt−yt∗\)\+o​\(11−αt\),f\(y\_\{t\},t\)=\-\\frac\{1\}\{1\-\\alpha\_\{t\}\}\\frac\{\\int\_\{G\(\\mathbb\{D\}\)\}\(y\-\\sqrt\{\\alpha\_\{t\}\}z\)\\mathcal\{N\}\(y\|\\sqrt\{\\alpha\_\{t\}\}z,\(1\-\\alpha\_\{t\}\)I\)\\mathrm\{d\}z\}\{\\int\_\{G\(\\mathbb\{D\}\)\}\\mathcal\{N\}\(y\|\\sqrt\{\\alpha\_\{t\}\}z,\(1\-\\alpha\_\{t\}\)I\)\\mathrm\{d\}z\}=\\frac\{1\}\{1\-\\alpha\_\{t\}\}\(y\_\{t\}\-y\_\{t\}^\{\*\}\)\+o\\left\(\\frac\{1\}\{1\-\\alpha\_\{t\}\}\\right\),\(32\)where the second equality is a corollary of Lemma[1](https://arxiv.org/html/2605.21692#Thmlemma1)to be justified later\. Then, hypothesis\(i​v\)\(iv\)implies thatγt​f​\(yt,t\)=∂tyt\+γt​yt\\gamma\_\{t\}f\(y\_\{t\},t\)=\\partial\_\{t\}y\_\{t\}\+\\gamma\_\{t\}y\_\{t\}is bounded, which in turn impliesyt−yt∗=\(1−αt\)​f​\(yt,t\)→0y\_\{t\}\-y^\{\*\}\_\{t\}=\(1\-\\alpha\_\{t\}\)f\(y\_\{t\},t\)\\rightarrow 0\. Sinceyt∗∈G​\(𝔻\)y^\{\*\}\_\{t\}\\in G\(\\mathbb\{D\}\), which is compact \(by hypothesis\(i​i\)\(ii\)and property of Lie groups\), andyty\_\{t\}converge \(by hypothesis\(i​v\)\(iv\)\), thenyt∗y\_\{t\}^\{\*\}converge andlimt→0yt=limt→0yt∗∈G​\(𝔻\)\\lim\_\{t\\rightarrow 0\}y\_\{t\}=\\lim\_\{t\\rightarrow 0\}y\_\{t\}^\{\*\}\\in G\(\\mathbb\{D\}\)\.

Therefore, we only need to prove the approximation in Eq\.[32](https://arxiv.org/html/2605.21692#A3.E32)\. Notingddthe dimension ofG​\(𝔻\)G\(\\mathbb\{D\}\),yt∗y\_\{t\}^\{\*\}the unique minimizer ofℓ​\(yt,G​\(𝔻\)\)\\ell\(y\_\{t\},G\(\\mathbb\{D\}\)\)\(by hypothesis\(i​i​i\)\(iii\)\), andI​\(t\)=∫G​\(𝔻\)\(y−αt​z\)​𝒩​\(y\|αt​z,\(1−αt\)​I\)​dzI\(t\)=\\int\_\{G\(\\mathbb\{D\}\)\}\(y\-\\sqrt\{\\alpha\_\{t\}\}z\)\\mathcal\{N\}\(y\|\\sqrt\{\\alpha\_\{t\}\}z,\(1\-\\alpha\_\{t\}\)I\)\\mathrm\{d\}z, we can write the following\.

I​\(t\)−\(yt−αt​yt∗\)​\(2​π​βt\)d/2\\displaystyle I\(t\)\-\(y\_\{t\}\-\\sqrt\{\\alpha\_\{t\}\}y^\{\*\}\_\{t\}\)\(2\\pi\\beta\_\{t\}\)^\{d/2\}=∫G​\(𝔻\)\(y−αt​z\)​𝒩​\(y\|αt​z,\(1−αt\)​I\)​dz\\displaystyle=\\int\_\{G\(\\mathbb\{D\}\)\}\(y\-\\sqrt\{\\alpha\_\{t\}\}z\)\\mathcal\{N\}\(y\|\\sqrt\{\\alpha\_\{t\}\}z,\(1\-\\alpha\_\{t\}\)I\)\\mathrm\{d\}z−∫G​\(𝔻\)\(y−αt​y∗\)​𝒩​\(y\|αt​z,\(1−αt\)​I\)​dz\\displaystyle\-\\int\_\{G\(\\mathbb\{D\}\)\}\(y\-\\sqrt\{\\alpha\_\{t\}\}y^\{\*\}\)\\mathcal\{N\}\(y\|\\sqrt\{\\alpha\_\{t\}\}z,\(1\-\\alpha\_\{t\}\)I\)\\mathrm\{d\}z=αt​∫G​\(𝔻\)\(y∗−z\)​𝒩​\(y\|αt​z,\(1−αt\)​I\)​dz\\displaystyle=\\sqrt\{\\alpha\_\{t\}\}\\int\_\{G\(\\mathbb\{D\}\)\}\(y^\{\*\}\-z\)\\mathcal\{N\}\(y\|\\sqrt\{\\alpha\_\{t\}\}z,\(1\-\\alpha\_\{t\}\)I\)\\mathrm\{d\}z‖I​\(t\)−\(yt−αt​yt∗\)​\(2​π​βt\)d/2‖\\displaystyle\\\|I\(t\)\-\(y\_\{t\}\-\\sqrt\{\\alpha\_\{t\}\}y^\{\*\}\_\{t\}\)\(2\\pi\\beta\_\{t\}\)^\{d/2\}\\\|≤αt​∫G​\(𝔻\)‖y∗−z‖​𝒩​\(y\|αt​z,\(1−αt\)​I\)​dz\\displaystyle\\leq\\sqrt\{\\alpha\_\{t\}\}\\int\_\{G\(\\mathbb\{D\}\)\}\\\|y^\{\*\}\-z\\\|\\mathcal\{N\}\(y\|\\sqrt\{\\alpha\_\{t\}\}z,\(1\-\\alpha\_\{t\}\)I\)\\mathrm\{d\}zMoreover, the functionz↦‖y∗−z‖z\\mapsto\\\|y^\{\*\}\-z\\\|is bounded, continuous and non\-negative onG​\(𝔻\)G\(\\mathbb\{D\}\), so that the conditions of Lemma[1](https://arxiv.org/html/2605.21692#Thmlemma1)are satisfied\. Therefore, we deduce by bounding thatI​\(t\)−\(yt−αt​yt∗\)​\(2​π​βt\)d/2=o​\(βtd/2\)I\(t\)\-\(y\_\{t\}\-\\sqrt\{\\alpha\_\{t\}\}y^\{\*\}\_\{t\}\)\(2\\pi\\beta\_\{t\}\)^\{d/2\}=o\(\\beta\_\{t\}^\{d/2\}\), which entailsI​\(t\)=\(yt−yt∗\)​\(2​π​βt\)d/2\+o​\(βtd/2\)I\(t\)=\(y\_\{t\}\-y^\{\*\}\_\{t\}\)\(2\\pi\\beta\_\{t\}\)^\{d/2\}\+o\(\\beta\_\{t\}^\{d/2\}\)\.

On the other side, we also deduce from Lemma[1](https://arxiv.org/html/2605.21692#Thmlemma1)that∫G​\(𝔻\)𝒩​\(y\|αt​z,\(1−αt\)​I\)​dz=\(2​π​βt\)d/2\+o​\(βtd/2\)\\int\_\{G\(\\mathbb\{D\}\)\}\\mathcal\{N\}\(y\|\\sqrt\{\\alpha\_\{t\}\}z,\(1\-\\alpha\_\{t\}\)I\)\\mathrm\{d\}z=\(2\\pi\\beta\_\{t\}\)^\{d/2\}\+o\(\\beta\_\{t\}^\{d/2\}\)\. Therefore, we have

f​\(yt,t\)=11−αt​\(yt−yt∗\)​\(2​π​βt\)d/2\+o​\(βtd/2\)\(2​π​βt\)d/2\+o​\(βtd/2\)=11−αt​\(yt−yt∗\)\+o​\(11−αt\)\.f\(y\_\{t\},t\)=\\frac\{1\}\{1\-\\alpha\_\{t\}\}\\frac\{\(y\_\{t\}\-y\_\{t\}^\{\*\}\)\(2\\pi\\beta\_\{t\}\)^\{d/2\}\+o\(\\beta\_\{t\}^\{d/2\}\)\}\{\(2\\pi\\beta\_\{t\}\)^\{d/2\}\+o\(\\beta\_\{t\}^\{d/2\}\)\}=\\frac\{1\}\{1\-\\alpha\_\{t\}\}\(y\_\{t\}\-y\_\{t\}^\{\*\}\)\+o\\left\(\\frac\{1\}\{1\-\\alpha\_\{t\}\}\\right\)\\;\.This shows thatΩf⊂G​\(𝔻\)\\Omega\_\{f\}\\subset G\(\\mathbb\{D\}\)\. For the reverse inclusion, we will use the assumption that each pointz∈𝔻z\\in\\mathbb\{D\}is a fixed point of the modelff\. More precisely, assume thatyt=g​\(z\)∈𝔻y\_\{t\}=g\(z\)\\in\\mathbb\{D\}withg∈Gg\\in Gandz∈𝔻z\\in\\mathbb\{D\}\. Then∂tyt=−γt\(g\(z\)−f\(g\(z\),t\)=−γTg\(z−f\(z,t\)\)=0\\partial\_\{t\}y\_\{t\}=\-\\gamma\_\{t\}\(g\(z\)\-f\(g\(z\),t\)=\-\\gamma\_\{T\}g\(z\-f\(z,t\)\)=0by equivariance offfand by the fixed point hypothesis\. Therefore, a trajectory starting atyT∈G​\(𝔻\)y\_\{T\}\\in G\(\\mathbb\{D\}\)stays atyTy\_\{T\}, which is hence a limit point\.

This establishesΩf=G​\(𝔻\)\\Omega\_\{f\}=G\(\\mathbb\{D\}\)and concludes the proof of Proposition[4](https://arxiv.org/html/2605.21692#Thmprop4)\. ∎

Proposition[4](https://arxiv.org/html/2605.21692#Thmprop4)establishes that an equivariant diffusion modelffgenerates samples inG​\(𝔻\)G\(\\mathbb\{D\}\)\. Therefore, we can identify its prediction spaceΩf\\Omega\_\{f\}withG​\(𝔻\)G\(\\mathbb\{D\}\)\. If the symmetry groupGGenforced by the architecture is aligned with the symmetries of the manifoldΩ\\Omega, then the effective dimension of the learning problem is reduced fromdΩd\_\{\\Omega\}todΩ/Gd\_\{\\Omega/G\}\.

###### Proposition 5\(Representation gap for an equivariant function\)\.

Assume thatΩ\\Omegais a compactdΩd\_\{\\Omega\}\-dimensional Riemannian manifold without boundary andppis continuous and strictly positive onΩ\\Omega\. Assume further thatGGis a Lie group of isometries acting smoothly, freely and properly onΩ\\Omega, and the orbitsG​\(y\)G\(y\)have the same Riemannian volume\|G\|\|G\|for each pointy∈Ωy\\in\\Omega\. Finally, assume thatffis an equivariant model satisfyingΩf=G​\(𝔻\)\\Omega\_\{f\}=G\(\\mathbb\{D\}\)\. Then the representation gap offfis

\(i\.i\.d\.\)ℛn\\displaystyle\\text\{\(i\.i\.d\.\)\}\\quad\\mathcal\{R\}\_\{n\}∼ℙ\|G\|​JdΩ/G​𝒱dΩ/Gn2/dΩ/G\(optimal\)ℛn⋆∼\|G\|​JdΩ/G∗​𝒱dΩ/G∗n2/dΩ/G\\displaystyle\\sim\_\{\\mathbb\{P\}\}\\frac\{\|G\|J\_\{d\_\{\\Omega/G\}\}\\mathcal\{V\}\_\{d\_\{\\Omega/G\}\}\}\{n^\{2/d\_\{\\Omega/G\}\}\}\\qquad\\qquad\\text\{\(optimal\)\}\\quad\\mathcal\{R\}\_\{n\}^\{\\star\}\\sim\\frac\{\|G\|J\_\{d\_\{\\Omega/G\}\}^\{\*\}\\mathcal\{V\}\_\{d\_\{\\Omega/G\}\}^\{\*\}\}\{n^\{2/d\_\{\\Omega/G\}\}\}\(33\)where the constants are computed with respect to the quotient metricℓΩ/G\\ell\_\{\\Omega/G\}onΩ/G\\Omega/G\.

###### Proof\.

The idea is to apply the Fubini theorem to factorize the integration over each orbit\. We haveΩf=G​\(𝔻\)\\Omega\_\{f\}=G\(\\mathbb\{D\}\)\. Therefore, using the quotient decomposition of the Riemannian measure and the fact that the action ofGGis isometric with constant\-volume orbits \(see for instanceGallotet al\.\([1990](https://arxiv.org/html/2605.21692#bib.bib96)\)\), we obtain

ℛ​\(Ω,Ωf\)=∫Ωminz∈G​\(𝔻\)⁡ℓ​\(y,z\)​p​\(y\)​dy=\|G\|​∫Ω/Gminz∈𝔻⁡ℓΩ/G​\(y,z\)​p​\(y\)​dy,\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)=\\int\_\{\\Omega\}\\min\_\{z\\in G\(\\mathbb\{D\}\)\}\\ell\(y,z\)p\(y\)\\mathrm\{d\}y=\|G\|\\int\_\{\\Omega/G\}\\min\_\{z\\in\\mathbb\{D\}\}\\ell\_\{\\Omega/G\}\(y,z\)p\(y\)\\mathrm\{d\}y,whereppnow denotes the induced density on the quotient spaceΩ/G\\Omega/G\. Therefore, we are in the setting of Propositions[2](https://arxiv.org/html/2605.21692#Thmprop2)and[3](https://arxiv.org/html/2605.21692#Thmprop3), sinceΩ/G\\Omega/Gis a manifold andℓΩ/G\\ell\_\{\\Omega/G\}is the quotient metric onΩ/G\\Omega/Gandd​y\\mathrm\{d\}yis the induced Riemannian measure onΩ/G\\Omega/G\. We can then conclude

ℛn∼ℙ\|G\|​JdΩ/G​𝒱dΩ/Gn2/dΩ/G,\\mathcal\{R\}\_\{n\}\\sim\_\{\\mathbb\{P\}\}\\frac\{\|G\|J\_\{d\_\{\\Omega/G\}\}\\mathcal\{V\}\_\{d\_\{\\Omega/G\}\}\}\{n^\{2/d\_\{\\Omega/G\}\}\}\\;,and likewise

ℛn∗​\|G\|​JdΩ/G∗​𝒱dΩ/G∗n2/dΩ/G\.\\mathcal\{R\}^\{\*\}\_\{n\}\\frac\{\|G\|J\_\{d\_\{\\Omega/G\}\}^\{\*\}\\mathcal\{V\}\_\{d\_\{\\Omega/G\}\}^\{\*\}\}\{n^\{2/d\_\{\\Omega/G\}\}\}\\;\.∎

Proposition[5](https://arxiv.org/html/2605.21692#Thmprop5)features an asymptotic evolution similar to the general case described in Propositions[2](https://arxiv.org/html/2605.21692#Thmprop2)and[3](https://arxiv.org/html/2605.21692#Thmprop3)\. In particular, we recover these formulas respectively when the groupGGcontains only the identity\.

## Appendix DConditional tasks

### D\.1Discrete\-class conditioning

We now extend these results to the more general case of conditional tasks\. BothΩ\\Omegaand𝔻\\mathbb\{D\}are subsets of𝒳×𝒴\\mathcal\{X\}\\times\\mathcal\{Y\}\. Let us first focus on the case whereΩ𝒳\\Omega\_\{\\mathcal\{X\}\}is finite and covered by the input dataset𝔻𝒳\\mathbb\{D\}\_\{\\mathcal\{X\}\}\. It is clear that for each inputx∈𝔻𝒳x\\in\\mathbb\{D\}\_\{\\mathcal\{X\}\}, the Propositions[2](https://arxiv.org/html/2605.21692#Thmprop2),[3](https://arxiv.org/html/2605.21692#Thmprop3)and[5](https://arxiv.org/html/2605.21692#Thmprop5)apply to the conditional dataset𝔻x\\mathbb\{D\}\_\{x\}and the conditional manifoldΩx\\Omega\_\{x\}\. We summarize this observation in the following Proposition\.

###### Proposition 6\(Representation gap for discrete conditional generation\)\.

Assume that the input spaceΩ𝒳\\Omega\_\{\\mathcal\{X\}\}is finite and that𝔻𝒳=Ω𝒳\\mathbb\{D\}\_\{\\mathcal\{X\}\}=\\Omega\_\{\\mathcal\{X\}\}\. Assume further that each conditional manifoldΩx\\Omega\_\{x\}is a compact Riemannian manifold without boundary, with common dimensiondΩd\_\{\\Omega\}, and conditional densitypxp\_\{x\}that is continuous and strictly positive onΩx\\Omega\_\{x\}\. For eachx∈Ω𝒳x\\in\\Omega\_\{\\mathcal\{X\}\}, assume thatGGis a Lie group of isometries acting smoothly, freely and properly onΩx\\Omega\_\{x\}, and that the orbitsG​\(y\)G\(y\)have constant Riemannian volume\|G\|\|G\|for each pointy∈Ωxy\\in\\Omega\_\{x\}\. Finally, assume thatffis an equivariant model satisfyingΩf=G​\(𝔻\)\\Omega\_\{f\}=G\(\\mathbb\{D\}\)\. Then the representation gap offfsatisfies

\(i\.i\.d\.\)ℛn\\displaystyle\\text\{\(i\.i\.d\.\)\}\\quad\\mathcal\{R\}\_\{n\}∼ℙ\|G\|n2/dΩ/G​∑x∈Ω𝒳Jx​𝒱x\(optimal\)ℛn⋆∼\|G\|n2/dΩ/G​∑x∈Ω𝒳Jx∗​𝒱x∗\\displaystyle\\sim\_\{\\mathbb\{P\}\}\\frac\{\|G\|\}\{n^\{2/d\_\{\\Omega/G\}\}\}\\sum\_\{x\\in\\Omega\_\{\\mathcal\{X\}\}\}J\_\{x\}\\mathcal\{V\}\_\{x\}\\qquad\\qquad\\text\{\(optimal\)\}\\quad\\mathcal\{R\}\_\{n\}^\{\\star\}\\,\\sim\\,\\frac\{\|G\|\}\{n^\{2/d\_\{\\Omega/G\}\}\}\\sum\_\{x\\in\\Omega\_\{\\mathcal\{X\}\}\}J\_\{x\}^\{\*\}\\mathcal\{V\}\_\{x\}^\{\*\}\(34\)whereJxJ\_\{x\}and𝒱x\\mathcal\{V\}\_\{x\}are computed on the quotient manifoldΩx/G\\Omega\_\{x\}/Gwith respect to the quotient metric induced byℓ\\ell\.

###### Proof\.

SinceΩ𝒳\\Omega\_\{\\mathcal\{X\}\}is finite, the representation gap decomposes as a finite sum over conditional manifolds:

ℛ​\(Ω,Ωf\)=∑x∈Ω𝒳ℛ​\(Ωx,\(Ωf\)x\)\.\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)=\\sum\_\{x\\in\\Omega\_\{\\mathcal\{X\}\}\}\\mathcal\{R\}\(\\Omega\_\{x\},\(\\Omega\_\{f\}\)\_\{x\}\)\.We conclude by applying Proposition[5](https://arxiv.org/html/2605.21692#Thmprop5)independently on each conditional manifoldΩx\\Omega\_\{x\}\. ∎

Note that Proposition[6](https://arxiv.org/html/2605.21692#Thmprop6)naturally generalizes to the setting where the conditional manifoldsΩx\\Omega\_\{x\}have different dimensionsdxd\_\{x\}for eachx∈Ω𝒳x\\in\\Omega\_\{\\mathcal\{X\}\}\. In this case, the representation gap is determined by the conditional manifolds with the highest dimension\. In particular, the intrinsic dimension becomesd=maxx∈Ω𝒳⁡dxd=\\max\_\{x\\in\\Omega\_\{\\mathcal\{X\}\}\}d\_\{x\}\.

### D\.2Continuous conditioning

We now turn to the case whereΩ𝒳\\Omega\_\{\\mathcal\{X\}\}is continuous\. Clearly, we require some result on howffbehaves outside the training data𝔻𝒳\\mathbb\{D\}\_\{\\mathcal\{X\}\}\. We assume thatffis Lipschitz with constantLL, which is a standard hypothesis in neural network analysis\.

We focus on supervised prediction\. Each inputx∈Ω𝒳x\\in\\Omega\_\{\\mathcal\{X\}\}is associated with a unique targety​\(x\)∈Ω𝒴y\(x\)\\in\\Omega\_\{\\mathcal\{Y\}\}, so that the observation manifoldΩ⊂Ω𝒳×Ω𝒴\\Omega\\subset\\Omega\_\{\\mathcal\{X\}\}\\times\\Omega\_\{\\mathcal\{Y\}\}can be identified with the graph of the functiony:Ω𝒳→Ω𝒴y:\\Omega\_\{\\mathcal\{X\}\}\\rightarrow\\Omega\_\{\\mathcal\{Y\}\}\. In particular, the intrinsic dimension ofΩ\\Omegacoincides with that of the input manifold,i\.e\.,dΩ=dΩ𝒳d\_\{\\Omega\}=d\_\{\\Omega\_\{\\mathcal\{X\}\}\}, independently of the dimension of𝒴\\mathcal\{Y\}\. We further assume that the modelffgenerates a unique predictionf​\(x\)f\(x\)for each inputx∈Ω𝒳x\\in\\Omega\_\{\\mathcal\{X\}\}, so that the prediction manifoldΩf\\Omega\_\{f\}can similarly be identified with the graph offf\. In this context, the conditional representation gap is defined by

ℛ​\(Ω,Ωf\)=∫Ω𝒳minz′∈Ωf⁡ℓ​\(z,z′\)​p​\(x\)​𝑑x,\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)=\\int\_\{\\Omega\_\{\\mathcal\{X\}\}\}\\min\_\{z^\{\\prime\}\\in\\Omega\_\{f\}\}\\ell\(z,z^\{\\prime\}\)\\,p\(x\)\\,dx,\(35\)wherez=\(x,y​\(x\)\)∈Ωz=\(x,y\(x\)\)\\in\\Omegaandz′=\(x′,f​\(x′\)\)∈Ωfz^\{\\prime\}=\(x^\{\\prime\},f\(x^\{\\prime\}\)\)\\in\\Omega\_\{f\}\. We denote byℓ𝒳\\ell\_\{\\mathcal\{X\}\}andℓ𝒴\\ell\_\{\\mathcal\{Y\}\}the metrics induced byℓ\\ellon𝒳\\mathcal\{X\}and𝒴\\mathcal\{Y\}respectively\.

We now study how to generalize the result of Proposition[6](https://arxiv.org/html/2605.21692#Thmprop6)to the conditional setting with continuous conditioning\. It is unclear whether we can derive a clean asymptotic equivalent of the representation gap in this case, since the geometry ofΩ\\Omegabecomes critical due to the coupling between input and target\. However, the next Proposition introduces an upper bound that follows the form introduced in Propositions[2](https://arxiv.org/html/2605.21692#Thmprop2),[3](https://arxiv.org/html/2605.21692#Thmprop3),[5](https://arxiv.org/html/2605.21692#Thmprop5)and[6](https://arxiv.org/html/2605.21692#Thmprop6)\.

###### Proposition 7\(Conditional representation gap of an equivariant model\)\.

Assume thatΩ\\Omegais a compactdΩd\_\{\\Omega\}\-dimensional Riemannian manifold without boundary andppis continuous and strictly positive onΩ\\Omega\. Assume also thatGGis a Lie group of isometries acting smoothly, freely and properly onΩ𝒳\\Omega\_\{\\mathcal\{X\}\}, and that the orbitsG​\(x\)G\(x\)have constant Riemannian volume\|G\|\|G\|for each pointx∈𝒳x\\in\\mathcal\{X\}\. Assume thatℓ\\ellis additively separable\. Finally, assume thatffis an equivariant model satisfyingΩf=G​\(𝔻\)\\Omega\_\{f\}=G\(\\mathbb\{D\}\), which is furthermore L\-lipschitz with constantL\>0L\>0\. Then the representation gap satisfies

\(i\.i\.d\.\)ℛn=Oℙ​\(1n2/d\)\(optimal\)ℛn⋆=O​\(1n2/d\),\\begin\{aligned\} \\text\{\(i\.i\.d\.\)\}\\quad\\mathcal\{R\}\_\{n\}&=O\_\{\\mathbb\{P\}\}\\left\(\\frac\{1\}\{n^\{2/d\}\}\\right\)\\qquad\\qquad\\text\{\(optimal\)\}\\quad\\mathcal\{R\}\_\{n\}^\{\\star\}&=O\\left\(\\frac\{1\}\{n^\{2/d\}\}\\right\)\\end\{aligned\}\\;,\(36\)whereΩ𝒳/G\\Omega\_\{\\mathcal\{X\}\}/Gdenotes the quotient space ofΩ𝒳\\Omega\_\{\\mathcal\{X\}\}by the symmetry groupGG, andd=dΩ𝒳/Gd=d\_\{\\Omega\_\{\\mathcal\{X\}\}/G\}denotes the dimension ofΩ𝒳/G\\Omega\_\{\\mathcal\{X\}\}/G\.

###### Proof\.

Using the equivariance offf, we proceed as in the proof of Proposition[5](https://arxiv.org/html/2605.21692#Thmprop5)and reduce the representation gap to a quantization problem on the quotient manifoldΩ𝒳/G\\Omega\_\{\\mathcal\{X\}\}/G\. Then, letz=\(x,y​\(x\)\)∈Ωz=\(x,y\(x\)\)\\in\\Omegadenote a data sample with inputxx, letx^=argminx′∈𝔻𝒳​ℓ𝒳​\(x,x′\)\\hat\{x\}=\\mathrm\{argmin\}\_\{x^\{\\prime\}\\in\\mathbb\{D\}\_\{\\mathcal\{X\}\}\}\\ell\_\{\\mathcal\{X\}\}\(x,x^\{\\prime\}\)denote the nearest training input toxx, and letz^=\(x^,y​\(x^\)\)\\hat\{z\}=\(\\hat\{x\},y\(\\hat\{x\}\)\)denote the corresponding training sample\. Sinceffinterpolates the training dataset, we havef​\(x^\)=y​\(x^\)f\(\\hat\{x\}\)=y\(\\hat\{x\}\)\. Using the additive separability ofℓ\\elland the Lipschitz continuity offf,

ℓ​\(\(x,y​\(x\)\),\(x^,f​\(x^\)\)\)≤ℓ𝒳​\(x,x^\)\+ℓ𝒴​\(f​\(x\),f​\(x^\)\)≤\(1\+L\)​ℓ𝒳​\(x,x^\)\.\\ell\\big\(\(x,y\(x\)\),\(\\hat\{x\},f\(\\hat\{x\}\)\)\\big\)\\leq\\ell\_\{\\mathcal\{X\}\}\(x,\\hat\{x\}\)\+\\ell\_\{\\mathcal\{Y\}\}\(f\(x\),f\(\\hat\{x\}\)\)\\leq\(1\+L\)\\ell\_\{\\mathcal\{X\}\}\(x,\\hat\{x\}\)\\;\.
Thus,ℛ​\(Ω,Ωf\)≤\(1\+L\)​∫Ω𝒳minx′∈𝔻𝒳⁡ℓ𝒳​\(x,x′\)​p​\(x\)​𝑑x,\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)\\leq\(1\+L\)\\int\_\{\\Omega\_\{\\mathcal\{X\}\}\}\\min\_\{x^\{\\prime\}\\in\\mathbb\{D\}\_\{\\mathcal\{X\}\}\}\\ell\_\{\\mathcal\{X\}\}\(x,x^\{\\prime\}\)\\,p\(x\)\\,dx,and the result follows from Proposition[5](https://arxiv.org/html/2605.21692#Thmprop5)\. ∎

## Appendix ELink with related work

In this section, we clarify the relations of the concept introduced in this article with several related works\.

### E\.1Generalization error

A natural question is to relate the representation gapℛ​\(Ω,Ωf\)\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)to the generalization error\(Shalev\-Shwartz and Ben\-David,[2014](https://arxiv.org/html/2605.21692#bib.bib204)\), commonly used to characterize generalization\. We focus on the setting of prediction tasks, for which there is a widely accepted definition of the generalization error,ℰ=∫Ωℓ​\(y​\(x\),f​\(x\)\)​p​\(x\)​dx\\mathcal\{E\}=\\int\_\{\\Omega\}\\ell\(y\(x\),f\(x\)\)p\(x\)\\mathrm\{d\}x\.

###### Proposition 8\(Comparison with generalization error\)\.

If the modelffisLL\-Lipschitz andℓ\\ellis additively separable, we have

11\+L​ℰ≤ℛ​\(Ω,Ωf\)≤ℰ\.\\frac\{1\}\{1\+L\}\\mathcal\{E\}\\leq\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)\\leq\\mathcal\{E\}\\;\.\(37\)

###### Proof\.

In the supervised setting, the representation gap can be written

ℛ​\(Ω,Ωf\)=∫Ωinfx′ℓ​\(\(x,y​\(x\)\),\(x′,f​\(x′\)\)\)​p​\(x\)​d​x\.\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)=\\int\_\{\\Omega\}\\inf\_\{x^\{\\prime\}\}\\ell\\left\(\(x,y\(x\)\)\\,,\\,\(x^\{\\prime\},f\(x^\{\\prime\}\)\)\\right\)p\(x\)\\mathrm\{d\}x\.We can see thatℛ​\(Ω,Ωf\)≤ℰ\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)\\leq\\mathcal\{E\}\(due to the inf operator\)\. Moreover, theLL\-Lipschitz regularity offfimplies that deviations in the output space are controlled by deviations in the input space\. Using the additivity ofℓ\\elltogether with the Lipschitz bound, we obtain

ℓ𝒴​\(y​\(x\),f​\(x\)\)≤\(1\+L\)​infx′ℓ​\(\(x,y​\(x\)\),\(x′,f​\(x′\)\)\),\\ell\_\{\\mathcal\{Y\}\}\(y\(x\),f\(x\)\)\\leq\(1\+L\)\\;\\inf\_\{x^\{\\prime\}\}\\ell\\left\(\(x,y\(x\)\)\\,,\\,\(x^\{\\prime\},f\(x^\{\\prime\}\)\)\\right\),and we conclude by integrating overΩ\\Omega\. ∎

Combining Theorems[4](https://arxiv.org/html/2605.21692#Thmtheorem4)and[5](https://arxiv.org/html/2605.21692#Thmtheorem5), we obtainℰ=O​\(1/n2/dΩ\)\\mathcal\{E\}=O\\left\(1/n^\{2/d\_\{\\Omega\}\}\\right\)asn→\+∞n\\rightarrow\+\\infty, a result closely related toTahmasebi and Jegelka \([2023](https://arxiv.org/html/2605.21692#bib.bib196)\)\. Moreover,ℛ​\(Ω,Ωf\)=0\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)=0impliesf​\(x\)=yxf\(x\)=y\_\{x\}almost everywhere, and thereforeℰ=0\\mathcal\{E\}=0\. Generalization error and representation gap are therefore closely related\.

### E\.2Wasserstein distance

Wasserstein distancePeyréet al\.\([2019](https://arxiv.org/html/2605.21692#bib.bib211)\)is typically used to measure neural network generalization\(Theiset al\.,[2015](https://arxiv.org/html/2605.21692#bib.bib207)\)\. Interestingly, we can see that the representation gapℛ​\(Ω,Ωf\)\\mathcal\{R\}\(\\Omega,\\Omega\_\{f\}\)is a particular case of the Wasserstein distance𝒲​\(Ω,Ωf\)\\mathcal\{W\}\(\\Omega,\\Omega\_\{f\}\), where each point\(x,y\)∈Ω\(x,y\)\\in\\Omegais associated to the probability massp​\(x,y\)p\(x,y\)and each prediction pointz∈Ωfz\\in\\Omega\_\{f\}is associated to the mass of its Voronoi cell\.

## Appendix FCompute resources

All experiments were conducted on consumer\-grade hardware\. The main experiments were independently reproduced on Google Colab using the default free public configuration\. No specialized compute infrastructure or large\-scale GPU resources were required\.

## Appendix GLimitations

Our analysis relies on the following main assumptions\.

Asymptotic regime\. Our results are asymptotic in the dataset sizenn\. However, our experiments suggest that the asymptotic regime is reached relatively quickly in practice \(see Section[5\.1](https://arxiv.org/html/2605.21692#S5.SS1)\)\.

Model assumptions\. Our results on generative modeling focus on DDIM diffusion models\. While the analysis extends naturally to the broader class of linear Gaussian diffusion models — including DDPM, variance\-exploding, and variance\-preserving formulations — recent diffusion architectures may fall outside this framework\. Moreover, our analysis assumes exact equivariance constraints and fully optimized models\. These assumptions are motivated by common practices in deep learning, where models are often trained in the interpolation regime and equivariance is enforced directly at the architectural level\. Nonetheless, extending the theory to approximate equivariance or partially trained models would nevertheless be of significant interest\.

Geometric assumptions\. Our theory relies on several regularity assumptions on the data manifoldΩ\\Omegaand the symmetry groupGG, including compactness and smoothness of the group action\. While such assumptions are standard in geometric learning theory, real\-world datasets may only approximately satisfy them\. Nevertheless, our empirical results on real\-world datasets suggest that the theory remains informative beyond the idealized setting considered here\.

Furthermore, some assumptions could likely be relaxed without fundamentally changing the analysis\. For instance, the constant\-volume orbit assumption is mainly introduced to simplify the exposition\.

Supervised prediction\. In the setting of supervised prediction, we derive asymptotic bounds rather than precise asymptotic equivalents\. Unlike the generative setting, the geometry of the joint manifoldΩ\\Omegabecomes critical due to the coupling between inputs and targets, making a full asymptotic characterization more challenging\. A deeper study of this regime is an important direction for future work\.

## Appendix HBroader impact

This work is primarily theoretical and aims to improve the understanding of generalization, sample efficiency, and equivariance in modern machine learning systems\. By relating generalization behavior to the intrinsic geometry of data manifolds, our results may contribute to the development of more data\-efficient learning algorithms and better principled model design\.

In particular, improved sample efficiency could benefit applications where data collection is expensive or limited, such as scientific imaging, healthcare, or robotics\. More broadly, our analysis may help clarify the role of symmetries and geometric structure in deep learning systems\.

At the same time, advances in generative modeling and sample\-efficient learning may contribute to the development of more capable generative systems, including systems that could potentially be misused for synthetic media generation or large\-scale content production\. However, the present work does not introduce new generative architectures or deployment methods, and focuses instead on the theoretical understanding of existing approaches\.