变分图册自编码器与自监督坐标图选择

# Variational Atlas Autoencodersand Self-Supervised Chart Selection
Source: [https://arxiv.org/html/2610.33577](https://arxiv.org/html/2610.33577)
Arno Coomans Merlin Nimier\-David Eugene d’Eon Boris BonevNVIDIA Graz University of Technology, Austria

###### Abstract

Complex, high\-dimensional data lie on manifolds that a single global latent coordinate system cannot represent without distortion\. We introduce Variational Atlas Autoencoders, a mixture\-of\-charts framework that learns an atlas of local latent representations, each responsible for one region of the data manifold\. The central difficulty in learning an atlas is deciding which chart should be responsible for which data point without labels\. We show that this decision need not be a heuristic\. By treating the chart index as a latent variable, the evidence lower bound yields the training objective and casts chart assignment as posterior inference, valid for any routing distribution\. Within this framework, we propose*latent\-evidence routing*, which assigns a point to a chart in proportion to the evidence of its encoding under that chart’s latent prior\. This has a simple interpretation: the closer a point lands to the center of a chart’s latent space, and the more confidently it is placed there, the more likely it is to belong to that chart\. Chart assignment thus emerges as posterior inference, depends on the encoder alone, and is fully self\-supervised, so the charts discover structure in the data on their own, separating classes and subclasses such as distinct types of shoes or faces without access to labels\. We further recover an atlas from the trained charts by identifying their overlaps and transition maps, enabling continuous interpolation within and across charts that stays close to the data manifold rather than traversing unsupported regions of latent space\. Compared to capacity\-matched baselines, Variational Atlas Autoencoders yields better\-structured latent spaces and higher sample quality on image datasets including CelebA, while sampling requires evaluating only a single chart’s decoder\.

## 1Introduction

Latent spaces learned by autoencoders underpin much of modern generative modeling, from latent diffusion\([Rombach et al\., 2022](https://arxiv.org/html/2610.33577#bib.bib23);[Vahdat et al\., 2021](https://arxiv.org/html/2610.33577#bib.bib4)\)to manifold\-learning flows\([Brehmer and Cranmer, 2020](https://arxiv.org/html/2610.33577#bib.bib24)\)\. These models are only as good as the learned coordinate system they are expressed in, and those coordinates almost always form a single global representation\. Forcing all data into one representation is a design choice inherited from the autoencoder, not a property of the data\.

Under the manifold hypothesis, data concentrate near akk\-dimensional manifoldℳ⊂ℝd\\mathcal\{M\}\\subset\\mathbb\{R\}^\{d\}withk≪dk\\ll d\([Fefferman et al\., 2016](https://arxiv.org/html/2610.33577#bib.bib19);[Brown et al\., 2023](https://arxiv.org/html/2610.33577#bib.bib15)\), and an autoencoder can be viewed as an approximate*chart*ofℳ\\mathcal\{M\}: the encoder assigns coordinates to each data point, and the decoder maps them back\. However, a single chart cannot cover every manifold without tearing or distortion\. For example, a sphere cannot be flattened onto a plane, and data made of separate clusters, such as distinct classes, cannot be covered by one continuous latent space without filling the gaps between them\. Autoencoders thus resort to non\-smooth latent spaces whose interpolants decode to implausible samples\([Arvanitidis et al\., 2018](https://arxiv.org/html/2610.33577#bib.bib2);[Oring et al\., 2021](https://arxiv.org/html/2610.33577#bib.bib3)\)\. This is unavoidable for disconnected data: the image of a connected latent space is connected, so it must cross the gaps between components\.

Differential geometry resolves this with an*atlas*: charts whose domains coverℳ\\mathcal\{M\}, overlap, and are related by transition maps\. Chart autoencoders\([Schonsheck et al\., 2019](https://arxiv.org/html/2610.33577#bib.bib13)\)and mixtures of VAEs\([Ye and Bors, 2022](https://arxiv.org/html/2610.33577#bib.bib16);[Alberti et al\., 2024](https://arxiv.org/html/2610.33577#bib.bib20)\)move in this direction by giving each component its own encoder–decoder pair, but leave two problems open\.

![Refer to caption](https://arxiv.org/html/2610.33577v1/torus_v2.png)Figure 1:Variational Atlas Autoencoders specialize to meaningful subsets of the data manifold with no label supervision\.Left: a genus\-2 surface covered by four22D charts, colored by the chart claiming each region, shows that multiple charts avoid the distortion of a single global 2D latent space\.Right: generated samples per chart on three image datasets\. On MNIST\([LeCun et al\., 1998](https://arxiv.org/html/2610.33577#bib.bib10)\)and Fashion\-MNIST\([Xiao et al\., 2017](https://arxiv.org/html/2610.33577#bib.bib12)\)\(1616\-D latents\), a chart typically covers one class or a related pair; on CelebA\([Liu et al\., 2015](https://arxiv.org/html/2610.33577#bib.bib11)\)\(128128\-D latents\), charts split by facial features, illumination, and background\.Without labels, the model must decide theresponsibilityof each chart for each data point on the manifold\. We make this decision part of the probabilistic model by treating the chart index as a latent variable\. Responsibility then enters the ELBO as the variational posteriorq⁡\(c\|x\)q\(c\|x\), which we call the routing function, and since the bound holds for any choice ofq⁡\(c\|x\)q\(c\|x\), the question becomes which router trains best\. Routing by each chart’s ELBO\([Alberti et al\., 2024](https://arxiv.org/html/2610.33577#bib.bib20)\)ties assignment to how well a chart already reconstructs a point, which can favour charts that improve early\. On the other hand, separate latent spaces use disjoint coordinate systems, so interpolation is undefined across charts, limiting theconnectivityof the model\. Without a way to transition between charts, the model cannot interpolate between arbitrary points on the data manifold, hindering downstream applications expressed in this coordinate system, such as latent\-space diffusion\.

#### Contributions\.

Variational Atlas Autoencoders address both problems\. \(i\) We formulate a variational mixture of charts, each a complete VAE, in which the training objective follows from the ELBO\. \(ii\) We propose*latent\-evidence routing*, which assigns a point to a chart in proportion to the evidence of its encoding under that chart’s prior: the closer the encoding lies to the chart’s centre, and the more confident it is, the higher its responsibility\. It depends on the encoders alone, meaning it does not require reconstructions for routing\. It discovers classes and subclasses without labels, such as sandals and heeled shoes in Fashion\-MNIST\. \(iii\) We turn the trained charts into an atlas by fitting overlaps \(*portals*\) where the router ties and building a chart transition graph, enabling interpolation within and across charts that stays close to the data manifold\. \(iv\) On MNIST, Fashion\-MNIST, and CelebA, our method achieves the best sample FID among parameter\-matched VAE and VaDE baselines, while sampling requires only a single \(smaller\) decoder pass\.

## 2Related Work

#### Manifold learning\.

The manifold hypothesis posits that high\-dimensional data concentrate near a manifold of much lower intrinsic dimension\.[Fefferman et al\. \(2016\)](https://arxiv.org/html/2610.33577#bib.bib19)provide a rigorous framework for testing this hypothesis for sufficiently smooth manifolds\. For image data,[Brown et al\. \(2023\)](https://arxiv.org/html/2610.33577#bib.bib15)provide empirical evidence for a more general union\-of\-manifolds hypothesis: data concentrate near disjoint manifolds with potentially varying intrinsic dimensions\. They further show that incorporating this structure into model design improves classification and generation\.

These observations suggest that using a single continuous latent space is generally insufficient to represent the data manifold\.[Tanielian et al\. \(2020\)](https://arxiv.org/html/2610.33577#bib.bib14)quantify this phenomenon for GANs: covering separated data components involves generating low\-quality samples in the gaps between them\.[Batson et al\. \(2021\)](https://arxiv.org/html/2610.33577#bib.bib21)demonstrate the same issue in the context of anomaly detection, giving low\-dimensional examples where topological mismatch leads to reconstruction errors for otherwise ordinary data points\. Our use of multiple charts allows separate representations of disconnected components and local coordinates within components whose topology precludes a single global chart, with transition maps relating coordinates on overlaps\.

#### Latent geometry and interpolation\.

Geometry also matters for density estimation and interpolation\. Manifold\-learning flows\([Brehmer and Cranmer, 2020](https://arxiv.org/html/2610.33577#bib.bib24)\)jointly learn an embedded manifold and a tractable density on it, while[Arvanitidis et al\. \(2021\)](https://arxiv.org/html/2610.33577#bib.bib29)equip the ambient space with a Riemannian metric and pull it back through the generator to construct more meaningful interpolation paths\. Together, these works motivate separating local parameterization from global structure\. Although we do not aim to generate the shortest interpolation paths, our charts are well\-suited to both intra\-chart and cross\-chart interpolation along straight lines thanks to their local coverage of the data manifold\.

#### Multi\-chart and mixture models\.

Mixtures of local linear latent\-variable models\([Ghahramani and Hinton, 1996](https://arxiv.org/html/2610.33577#bib.bib28);[Tipping and Bishop, 1999](https://arxiv.org/html/2610.33577#bib.bib27)\)are the classical precursor to our model, with mixture responsibilities in the role of the routing posterior\.[Teh and Roweis \(2002\)](https://arxiv.org/html/2610.33577#bib.bib25)and[Brand \(2002\)](https://arxiv.org/html/2610.33577#bib.bib26)align such local models into one global embedding, whereas we keep the local coordinates and learn the transition maps between them \([Eq\. 8](https://arxiv.org/html/2610.33577#S3.E8)\)\.

In the nonlinear \(deep\) setting, variational deep embeddings \(VaDE\)\([Jiang et al\., 2017](https://arxiv.org/html/2610.33577#bib.bib17)\)place a Gaussian mixture prior on the latent space of a single VAE and read the cluster assignment from the mixture responsibilities\. Since all components share the encoder and decoder, this is still a single chart, which cannot cover most non\-trivial data manifolds \([Sec\. A\.2](https://arxiv.org/html/2610.33577#A1.SS2)\)\. Mixtures of VAEs with a separate encoder–decoder pair per component\([Ye and Bors, 2022](https://arxiv.org/html/2610.33577#bib.bib16);[Alberti et al\., 2024](https://arxiv.org/html/2610.33577#bib.bib20)\)remove this restriction and are the closest generative models to ours\. To route data to a specific chart,[Alberti et al\. \(2024\)](https://arxiv.org/html/2610.33577#bib.bib20)use the per\-component ELBO\. We show that this couples assignment to reconstruction quality and starves charts that start out worse, and we replace it with latent\-evidence routing, which depends on the encoder alone\. Neither work constructs overlaps or transition maps between components\.

The chart auto\-encoder \(CAE\) of[Schonsheck et al\. \(2019\)](https://arxiv.org/html/2610.33577#bib.bib13)uses several deterministic encoder–decoder pairs and shows that a single simply connected latent space cannot faithfully represent a manifold with non\-contractible topology\. Charts are assigned by a separate network trained toward a softmax over per\-chart reconstruction errors, and transition maps compose one chart’s decoder with another’s encoder, but the overlaps on which they should hold are never constructed\.[Stolberg\-Larsen and Sommer \(2022\)](https://arxiv.org/html/2610.33577#bib.bib6)unify such multi\-chart models as*atlas generative models*and interpolate across charts on a graph over encoded data points\. Two charts are linked only at individual data points whose routing exceeds a threshold in both, by connecting that point’s two encodings\. We differ in three respects: our chart assignment follows from the ELBO \([Eq\. 4](https://arxiv.org/html/2610.33577#S3.E4)\) and is computed from the chart encoders rather than a separate network; overlaps are portals, distributions over a chart’s latent space located where the router ties, rather than individual data points; and each portal comes with an explicit transition map, so charts can be crossed from anywhere within it \([Sec\. 3\.3](https://arxiv.org/html/2610.33577#S3.SS3)\)\.

## 3Methodology

The overall goal of a generative model is to represent a data distributionp⁡\(𝐱\)p\(\\mathbf\{x\}\)\. In a latent\-variable model, the distribution is often represented using a single continuous latent space with a simple prior, typically a multivariate standard normal distribution𝐳∼𝒩⁡\(𝟎,𝐈\)\\mathbf\{z\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\)\. The encoder must therefore represent all samples𝐱\\mathbf\{x\}from the data distribution within a single latent space, even when the data lie on a highly nonlinear or disconnected manifold\. Our proposed solution is to model the data distribution using multiple latent representations\. Each latent component can specialize in a particular subset, mode, or locally coherent region of the data manifold\.

![Refer to caption](https://arxiv.org/html/2610.33577v1/methodology-training.png)

Figure 2:Top: we trainCCencoder–decoder pairs to model the data manifold using independent charts\. Our training objective, the ELBO, decomposes into per\-chart losses weighted by a routing probability\. The KL divergence on the chart selection prior encourages all charts to be used with equal probability\.Bottom: sampling from a chart is then as simple as selecting a chartccuniformly at random, sampling a latent𝐳c\\mathbf\{z\}\_\{c\}from the standard normal prior, and decoding it\. Moment matching can optionally be added, see[Sec\. 3\.2](https://arxiv.org/html/2610.33577#S3.SS2)\.### 3\.1Variational Chart Selection

To modelpθ​\(𝐱\)p\_\{\\theta\}\(\\mathbf\{x\}\)using multiple charts, we introduce a discrete chart variablec∈\{1,…,C\}c\\in\\\{1,\\ldots,C\\\}\. As a modeling assumption, each chart is assigned equal prior probability, so thatp⁡\(c\)=1/Cp\(c\)=1/C\. Each chart is further equipped with a continuousLL\-dimensional latent variable𝐳c∈ℝL\\mathbf\{z\}\_\{c\}\\in\\mathbb\{R\}^\{L\}with conditional priorp⁡\(𝐳c∣c\)=𝒩⁡\(𝟎,𝐈\)p\(\\mathbf\{z\}\_\{c\}\\mid c\)=\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\)\. The resulting modeled data distribution is then given by:

pθ​\(𝐱\)=∑c=1Cp⁡\(c\)​∫pθ​\(𝐱∣𝐳c,c\)​p​\(𝐳c∣c\)​d​𝐳c\.p\_\{\\theta\}\(\\mathbf\{x\}\)=\\sum\_\{c=1\}^\{C\}p\(c\)\\int p\_\{\\theta\}\(\\mathbf\{x\}\\mid\\mathbf\{z\}\_\{c\},c\)\\,p\(\\mathbf\{z\}\_\{c\}\\mid c\)\\,\\mathrm\{d\}\\mathbf\{z\}\_\{c\}\.\(1\)Given an observation𝐱\\mathbf\{x\}, inference requires the posterior distribution over the chart assignmentpθ​\(c∣𝐱\)p\_\{\\theta\}\(c\\mid\\mathbf\{x\}\)and the chart\-specific latent posteriorpθ​\(𝐳c∣c,𝐱\)p\_\{\\theta\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\. Since these posterior distributions are generally intractable, we introduce the variational approximationsqϕ​\(c∣𝐱\)q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)andqϕ​\(𝐳c∣c,𝐱\)q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\. Together, they define the approximate posterior

qϕ​\(c,𝐳c∣𝐱\)=qϕ​\(c∣𝐱\)​qϕ​\(𝐳c∣c,𝐱\),q\_\{\\phi\}\(c,\\mathbf\{z\}\_\{c\}\\mid\\mathbf\{x\}\)=q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\\,q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\),\(2\)which approximates the true posteriorpθ​\(c,𝐳c∣𝐱\)p\_\{\\theta\}\(c,\\mathbf\{z\}\_\{c\}\\mid\\mathbf\{x\}\)\. Each chartccis realized by an encoderfc​\(𝐱\)=\(𝝁c​\(𝐱\),𝚺c​\(𝐱\)\)f\_\{c\}\(\\mathbf\{x\}\)=\(\\bm\{\\mu\}\_\{c\}\(\\mathbf\{x\}\),\\bm\{\\Sigma\}\_\{c\}\(\\mathbf\{x\}\)\), which parameterizes the diagonal Gaussianqϕ​\(𝐳c∣c,𝐱\)=𝒩⁡\(𝝁c​\(𝐱\),𝚺c​\(𝐱\)\)q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)=\\mathcal\{N\}\(\\bm\{\\mu\}\_\{c\}\(\\mathbf\{x\}\),\\bm\{\\Sigma\}\_\{c\}\(\\mathbf\{x\}\)\), and a decodergcg\_\{c\}, whose output𝐱^=gc​\(𝐳c\)\\hat\{\\mathbf\{x\}\}=g\_\{c\}\(\\mathbf\{z\}\_\{c\}\)is the mean ofpθ​\(𝐱∣𝐳c,c\)p\_\{\\theta\}\(\\mathbf\{x\}\\mid\\mathbf\{z\}\_\{c\},c\)\. Throughout,ϕ\\phiandθ\\thetadenote the parameters of all encoders and decoders, respectively\. Direct evaluation and optimization of the marginal log\-likelihood in[Eq\. 1](https://arxiv.org/html/2610.33577#S3.E1)are generally intractable because the integral over the continuous latent variables cannot usually be computed in closed form\. As in the single\-chart VAE setting\([Kingma and Welling, 2014](https://arxiv.org/html/2610.33577#bib.bib7)\), we therefore optimize a variational lower bound onlog⁡pθ​\(𝐱\)\\log p\_\{\\theta\}\(\\mathbf\{x\}\),

log⁡pθ​\(𝐱\)≥ℒELBO​\(𝐱\)=𝔼qϕ​\(c,𝐳c∣𝐱\)​\[log⁡pθ​\(𝐱,𝐳c,c\)qϕ​\(c,𝐳c∣𝐱\)\]\.\\log p\_\{\\theta\}\(\\mathbf\{x\}\)\\geq\\mathcal\{L\}\_\{\\mathrm\{ELBO\}\}\(\\mathbf\{x\}\)=\\mathbb\{E\}\_\{q\_\{\\phi\}\(c,\\mathbf\{z\}\_\{c\}\\mid\\mathbf\{x\}\)\}\\left\[\\log\\frac\{p\_\{\\theta\}\(\\mathbf\{x\},\\mathbf\{z\}\_\{c\},c\)\}\{q\_\{\\phi\}\(c,\\mathbf\{z\}\_\{c\}\\mid\\mathbf\{x\}\)\}\\right\]\.\(3\)Relating the lower bound to the exact posterior and expressing terms as KL divergences, we obtain

ℒELBO​\(𝐱\)=\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{ELBO\}\}\(\\mathbf\{x\}\)=−DKL\(qϕ\(c∣𝐱\)∥p\(c\)\)\\displaystyle\-D\_\{\\mathrm\{KL\}\}\\\!\\left\(q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\\,\\\|\\,p\(c\)\\right\)\(4\)\+∑c=1Cqϕ\(c∣𝐱\)\[𝔼qϕ​\(𝐳c∣c,𝐱\)\[logpθ\(𝐱∣𝐳c,c\)\]−DKL\(qϕ\(𝐳c∣c,𝐱\)∥p\(𝐳c∣c\)\)\]\.\\displaystyle\+\\sum\_\{c=1\}^\{C\}q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\\left\[\\mathbb\{E\}\_\{q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\}\\left\[\\log p\_\{\\theta\}\(\\mathbf\{x\}\\mid\\mathbf\{z\}\_\{c\},c\)\\right\]\-D\_\{\\mathrm\{KL\}\}\\\!\\left\(q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\\,\\\|\\,p\(\\mathbf\{z\}\_\{c\}\\mid c\)\\right\)\\right\]\.Writing the chart router as an explicit finite sum makes it clear that the per\-chart ELBO

ℒc\(𝐱\)=𝔼qϕ​\(𝐳c∣c,𝐱\)\[logpθ\(𝐱∣𝐳c,c\)\]−DKL\(qϕ\(𝐳c∣c,𝐱\)∥p\(𝐳c∣c\)\)\\mathcal\{L\}\_\{c\}\(\\mathbf\{x\}\)=\\mathbb\{E\}\_\{q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\}\\\!\\left\[\\log p\_\{\\theta\}\(\\mathbf\{x\}\\mid\\mathbf\{z\}\_\{c\},c\)\\right\]\-D\_\{\\mathrm\{KL\}\}\\\!\\left\(q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\\,\\\|\\,p\(\\mathbf\{z\}\_\{c\}\\mid c\)\\right\)\(5\)is simply weighted by each chart’s routing probabilityqϕ​\(c∣𝐱\)q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\), and the prior routing termDKL\(qϕ\(c∣𝐱\)∥p\(c\)\)D\_\{\\mathrm\{KL\}\}\(q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\\,\\\|\\,p\(c\)\)regularizes the routing function\. The role of each term is visualized in[Figure 2](https://arxiv.org/html/2610.33577#S3.F2)\. In practice, we minimize the weighted lossℓ⁡\(𝐱\)\\ell\(\\mathbf\{x\}\)of[Eq\. 17](https://arxiv.org/html/2610.33577#A2.E17), in whichℒrecon​\(𝐱,gc​\(𝐳c\)\)\\mathcal\{L\}\_\{\\mathrm\{recon\}\}\(\\mathbf\{x\},g\_\{c\}\(\\mathbf\{z\}\_\{c\}\)\)is either the reconstruction loss−log⁡pθ​\(𝐱∣𝐳c,c\)\-\\log p\_\{\\theta\}\(\\mathbf\{x\}\\mid\\mathbf\{z\}\_\{c\},c\)or a perceptual loss, and the two KL terms are weighted by hyperparametersβ\\betaandγ\\gamma\([Appendix B](https://arxiv.org/html/2610.33577#A2),[Appendix D](https://arxiv.org/html/2610.33577#A4)\)\.

![Refer to caption](https://arxiv.org/html/2610.33577v1/fig_routing.png)Figure 3:Illustration of latent\-evidence routing\. Two inputs \(left\) are encoded by every chart; the evidence of each encoded point under its chart’s latent prior induces a distribution over charts, which is used for routing\. For the histogram \(right\), the temperature parameter was artificially raised toT=10T=10to make the distribution easier to visualize\.The bound in[Eq\. 4](https://arxiv.org/html/2610.33577#S3.E4)holds for*any*routing distributionqϕ​\(c∣𝐱\)q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\), which leaves the choice of router as a design decision\.

#### ELBO routing\.

A natural choice is to route by the evidence,qϕ​\(c∣𝐱\)∝p⁡\(c\)​exp⁡\(ℒc​\(𝐱\)/T\)q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\\propto p\(c\)\\exp\\\!\\left\(\\mathcal\{L\}\_\{c\}\(\\mathbf\{x\}\)/T\\right\), whereTTis a temperature parameter that anneals the chart assignment\. Substituting this router into[Eq\. 4](https://arxiv.org/html/2610.33577#S3.E4)collapses the bound intoℒELBO​\(𝐱\)=T​log​∑cp⁡\(c\)​exp⁡\(ℒc​\(𝐱\)/T\)\\mathcal\{L\}\_\{\\mathrm\{ELBO\}\}\(\\mathbf\{x\}\)=T\\log\\sum\_\{c\}p\(c\)\\exp\\\!\\left\(\\mathcal\{L\}\_\{c\}\(\\mathbf\{x\}\)/T\\right\), which is tight atT=1T=1\. We refer to this variant as*ELBO routing*, following related work\([Alberti et al\., 2024](https://arxiv.org/html/2610.33577#bib.bib20)\)\.

#### Latent\-evidence routing\.

Instead, routing can be based on the latent prior alone, without involving the decoder\. We propose*latent\-evidence routing*, guided by the intuition that a chart’s routing probability should be proportional to the likelihood of the encoded point under that chart’s Gaussian prior\. All variants below take the formqϕ​\(c∣𝐱\)∝p⁡\(c\)​exp⁡\(sc​\(𝐱\)/T\)q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\\propto p\(c\)\\exp\\\!\\left\(s\_\{c\}\(\\mathbf\{x\}\)/T\\right\), a softmax over a per\-chart scorescs\_\{c\}with temperatureTT, and differ only in howscs\_\{c\}accounts for the posterior uncertainty𝚺c​\(𝐱\)=diag⁡\(σc,12,…,σc,L2\)\\bm\{\\Sigma\}\_\{c\}\(\\mathbf\{x\}\)=\\operatorname\{diag\}\(\\sigma\_\{c,1\}^\{2\},\\dots,\\sigma\_\{c,L\}^\{2\}\)\.[Figure 3](https://arxiv.org/html/2610.33577#S3.F3)provides a visual overview of latent\-evidence routing\.

The simplest score evaluates the prior at the posterior mean,sc​\(𝐱\)=log⁡p⁡\(𝐳c=𝝁c​\(𝐱\)∣c\)s\_\{c\}\(\\mathbf\{x\}\)=\\log p\(\\mathbf\{z\}\_\{c\}=\\bm\{\\mu\}\_\{c\}\(\\mathbf\{x\}\)\\mid c\), ignoring the posterior uncertainty entirely\. We refer to this as*μ\\mu\-based routing*\. Alternatively, the*expected log\-prior*score averages the log\-prior over the posterior,

sc​\(𝐱\)=𝔼qϕ​\(𝐳c∣c,𝐱\)​\[log⁡p⁡\(𝐳c∣c\)\]=−12​\(‖𝝁c‖2\+tr⁡𝚺c\)\+const\.\\displaystyle s\_\{c\}\(\\mathbf\{x\}\)=\\mathbb\{E\}\_\{q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\}\\\!\\left\[\\log p\(\\mathbf\{z\}\_\{c\}\\mid c\)\\right\]=\-\\frac\{1\}\{2\}\\left\(\\\|\\bm\{\\mu\}\_\{c\}\\\|^\{2\}\+\\operatorname\{tr\}\\bm\{\\Sigma\}\_\{c\}\\right\)\+\\mathrm\{const\}\.\(6\)A single\-sample Monte Carlo estimate of this expectation recovers the*resampling\-based routing*, at the cost of additional variance in the routing weights\.

Finally, we propose*prior\-marginal routing*, which scores each chart by the prior mass its posterior covers, i\.e\. the evidence of the posterior under the chart’s latent prior,

sc\(𝐱\)=log∫qϕ\(𝐳c∣c,𝐱\)p\(𝐳c∣c\)d𝐳c=−12\(∥𝝁c∥\(𝐈\+𝚺c\)−12\+logdet\(𝐈\+𝚺c\)\)\+const,\\displaystyle s\_\{c\}\(\\mathbf\{x\}\)=\\log\\int q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\\,p\(\\mathbf\{z\}\_\{c\}\\mid c\)\\,\\mathrm\{d\}\\mathbf\{z\}\_\{c\}=\-\\frac\{1\}\{2\}\\left\(\\\|\\bm\{\\mu\}\_\{c\}\\\|^\{2\}\_\{\(\\mathbf\{I\}\+\\bm\{\\Sigma\}\_\{c\}\)^\{\-1\}\}\+\\log\\det\(\\mathbf\{I\}\+\\bm\{\\Sigma\}\_\{c\}\)\\right\)\+\\mathrm\{const\},\(7\)where‖𝝁c‖\(𝐈\+𝚺c\)−12=𝝁c⊤​\(𝐈\+𝚺c\)−1​𝝁c\\\|\\bm\{\\mu\}\_\{c\}\\\|^\{2\}\_\{\(\\mathbf\{I\}\+\\bm\{\\Sigma\}\_\{c\}\)^\{\-1\}\}=\\bm\{\\mu\}\_\{c\}^\{\\top\}\(\\mathbf\{I\}\+\\bm\{\\Sigma\}\_\{c\}\)^\{\-1\}\\bm\{\\mu\}\_\{c\}\. This is the marginal likelihood𝒩⁡\(𝝁c​\(𝐱\),0,𝐈\+𝚺c​\(𝐱\)\)\\mathcal\{N\}\(\\bm\{\\mu\}\_\{c\}\(\\mathbf\{x\}\);\\,\\mathbf\{0\},\\,\\mathbf\{I\}\+\\bm\{\\Sigma\}\_\{c\}\(\\mathbf\{x\}\)\)of the encoder output regarded as a noisy observation of a latent𝐳c∼p⁡\(𝐳c∣c\)\\mathbf\{z\}\_\{c\}\\sim p\(\\mathbf\{z\}\_\{c\}\\mid c\)with observation noise𝚺c​\(𝐱\)\\bm\{\\Sigma\}\_\{c\}\(\\mathbf\{x\}\)\. All three scores are available in closed form for diagonal Gaussian posteriors\. Whereas ELBO routing measures the evidence of𝐱\\mathbf\{x\}in data space and therefore depends on the decoder,scs\_\{c\}measures the evidence of the encoded point in latent space and depends on the encoder only\. It formalizes the latent\-centrality intuition of[Sec\. 1](https://arxiv.org/html/2610.33577#S1): a point belongs to the chart whose posterior places it closest to the center of the chart’s latent domain\.

The three scores differ only in how they treat uncertainty, and in the limit𝚺c→𝟎\\bm\{\\Sigma\}\_\{c\}\\to\\mathbf\{0\}, all of them coincide withμ\\mu\-based routing\. The expected log\-prior \([Eq\. 6](https://arxiv.org/html/2610.33577#S3.E6)\) is a lower bound on the prior marginal by Jensen’s inequality, but it penalizes the full Euclidean distance‖𝝁c‖2\\\|\\bm\{\\mu\}\_\{c\}\\\|^\{2\}regardless of the posterior spread and merely adds a variance penaltytr⁡𝚺c\\operatorname\{tr\}\\bm\{\\Sigma\}\_\{c\}\. The prior marginal of[Eq\. 7](https://arxiv.org/html/2610.33577#S3.E7)instead measures this distance in the Mahalanobis norm of𝐈\+𝚺c\\mathbf\{I\}\+\\bm\{\\Sigma\}\_\{c\}, so that uncertain encodings are penalized less\.

#### Impact on training\.

The choice of router also affects training: since[Eq\. 4](https://arxiv.org/html/2610.33577#S3.E4)is a valid lower bound for anyqϕ​\(c∣𝐱\)q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\), we substitute the router in and maximizeℒELBO\\mathcal\{L\}\_\{\\mathrm\{ELBO\}\}directly, so the encoders receive gradients through both the per\-chart boundsℒc\\mathcal\{L\}\_\{c\}and the routing weights\. Under ELBO routing, a chart’s weight is determined by how well it already reconstructs𝐱\\mathbf\{x\}\. Charts that start out worse receive smaller weights, hence smaller gradients, and fall further behind\. Latent\-evidence routing removes this feedback loop by assigning points according to where they land in latent space rather than how well they are reconstructed, and the KL term in[Eq\. 4](https://arxiv.org/html/2610.33577#S3.E4)keeps the routing weights from collapsing onto a single chart\. We adopt prior\-marginal routing \([Eq\. 7](https://arxiv.org/html/2610.33577#S3.E7)\) as our default due to its robustness and report ablation results for all variants in[Table 5](https://arxiv.org/html/2610.33577#A6.T5)in the appendix\.

### 3\.2Sampling

#### Moment matching \(optional\)\.

After training, the aggregate posteriorqϕ​\(𝐳c∣c\)=𝔼p⁡\(𝐱\)​qϕ​\(𝐳c∣c,𝐱\)q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c\)=\\mathbb\{E\}\_\{p\(\\mathbf\{x\}\)\}\\,q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)of a chart need not coincide exactly with its prior𝒩⁡\(𝟎,𝐈\)\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\), and sampling from regions of the prior that carry no posterior mass produces poor samples\([Hoffman and Johnson, 2016](https://arxiv.org/html/2610.33577#bib.bib22)\)\. We therefore optionally replace the prior by the moment\-matched aggregate posteriorp~​\(𝐳c∣c\):=𝒩⁡\(𝐦c,𝐒c\)\\tilde\{p\}\(\\mathbf\{z\}\_\{c\}\\mid c\):=\\mathcal\{N\}\(\\mathbf\{m\}\_\{c\},\\mathbf\{S\}\_\{c\}\), whose mean and covariance are those ofqϕ​\(𝐳c∣c\)q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c\)estimated once over the training set\. Without moment matching, we usep~=p\\tilde\{p\}=p\. The chart prior is treated likewise: the uniformp⁡\(c\)=1/Cp\(c\)=1/Cis replaced by the empirical routing massp^​\(c\)=𝔼p⁡\(𝐱\)​qϕ​\(c∣𝐱\)\\hat\{p\}\(c\)=\\mathbb\{E\}\_\{p\(\\mathbf\{x\}\)\}\\,q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\), so that charts claiming more of the training data are sampled more often\. Both corrections are cheap, require no retraining, and are applied in all sampling experiments unless otherwise stated\.

#### Unconditional sampling\.

Generation follows Algorithm[2](https://arxiv.org/html/2610.33577#alg2): draw a chartc∼p^​\(c\)c\\sim\\hat\{p\}\(c\)and a latent𝐳c∼p~​\(𝐳c∣c\)\\mathbf\{z\}\_\{c\}\\sim\\tilde\{p\}\(\\mathbf\{z\}\_\{c\}\\mid c\), and decode𝐱^=gc​\(𝐳c\)\\hat\{\\mathbf\{x\}\}=g\_\{c\}\(\\mathbf\{z\}\_\{c\}\)\. Only the selected decoder is evaluated, so the sampling cost is that of a single VAE with the same backbone\. Sampling within one chart yields samples from the corresponding region of the data manifold \([Figure 9](https://arxiv.org/html/2610.33577#A6.F9)\), which makes the chart index an interpretable, label\-free handle on the generated content\.

#### Encoding\.

In contrast, encoding a data point requires allCCencoders to evaluate the routing weightsqϕ​\(c∣𝐱\)q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\. For applications where encoding cost matters, one may train a lightweight classifier to predict the routing distribution so that only the relevant encoder is evaluated\.

### 3\.3Overlaps, Transition Maps, and Portals

An atlas on a manifold is a family of charts whose domains cover the manifold and whose*transition maps*are homeomorphisms on the overlaps \([Sec\. A\.1](https://arxiv.org/html/2610.33577#A1.SS1)\)\. The model described in[Sec\. 3\.1](https://arxiv.org/html/2610.33577#S3.SS1)supplies coordinate maps, but the routing posteriorqϕ​\(c∣𝐱\)q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)induces a*partition*of the data, and a partition has no overlaps\. Here, we describe how overlaps and transition maps between charts can be constructed to recover an atlas and thereby enable movement from any point on the manifold to any other\.

#### Transition map\.

The decodergcg\_\{c\}and the encoder mean𝝁c\\bm\{\\mu\}\_\{c\}play the roles of the parametrization and the coordinate map of chartcc\([Sec\. A\.3](https://arxiv.org/html/2610.33577#A1.SS3)\), so the transition map from chartaato chartbband its round trip are

ψa​b​\(𝐳\)=𝝁b​\(ga​\(𝐳\)\),Ra​b=ψb​a∘ψa​b\.\\psi\_\{ab\}\(\\mathbf\{z\}\)=\\bm\{\\mu\}\_\{b\}\\bigl\(g\_\{a\}\(\\mathbf\{z\}\)\\bigr\),\\qquad R\_\{ab\}=\\psi\_\{ba\}\\circ\\psi\_\{ab\}\.\(8\)The atlas condition on the overlap isRa​b=idR\_\{ab\}=\\mathrm\{id\}\. We measure latent distances in the Mahalanobis metric of the sampling distributionp~​\(𝐳c∣c\)=𝒩⁡\(𝐦c,𝐒c\)\\tilde\{p\}\(\\mathbf\{z\}\_\{c\}\\mid c\)=\\mathcal\{N\}\(\\mathbf\{m\}\_\{c\},\\mathbf\{S\}\_\{c\}\)of[Sec\. 3\.2](https://arxiv.org/html/2610.33577#S3.SS2),‖𝐮‖c2=1L​𝐮⊤​𝐒c−1​𝐮\\\|\\mathbf\{u\}\\\|\_\{c\}^\{2\}=\\tfrac\{1\}\{L\}\\,\\mathbf\{u\}^\{\\top\}\\mathbf\{S\}\_\{c\}^\{\-1\}\\mathbf\{u\}, normalised so that a typical latent of chartcchas‖𝝁c​\(𝐱\)−𝐦c‖c2≈1\\\|\\bm\{\\mu\}\_\{c\}\(\\mathbf\{x\}\)\-\\mathbf\{m\}\_\{c\}\\\|\_\{c\}^\{2\}\\approx 1\.

#### Portals as variational overlaps\.

An overlap is a region of the data manifold that*both*charts claim, and the routerqϕ​\(c∣𝐱\)q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)is the model’s own statement of which chart claims a point\. We therefore represent the overlap of the ordered pair\(a,b\)\(a,b\)by a Gaussianqa​b​\(𝐳\)=𝒩⁡\(𝐦a​b,𝐒a​b\)q\_\{ab\}\(\\mathbf\{z\}\)=\\mathcal\{N\}\(\\mathbf\{m\}\_\{ab\},\\mathbf\{S\}\_\{ab\}\)on the latent space of chartaa, with𝐒a​b\\mathbf\{S\}\_\{ab\}parameterized by its Cholesky factor, fitted to lie where the router assigns equal probability toaaandbb\(a tie\)\. We refer to these overlaps as “portals”, as they will be used to transition across charts\. The encoders’ and decoders’ weights are kept frozen while optimizing portal locations\. Writing𝐱=ga​\(𝐳\)\\mathbf\{x\}=g\_\{a\}\(\\mathbf\{z\}\),\(Ra​b\)\#​qa​b\(R\_\{ab\}\)\_\{\\\#\}q\_\{ab\}for the pushforward ofqa​bq\_\{ab\}under the round tripRa​bR\_\{ab\}, andmm⁡\(⋅\)\\mathrm\{mm\}\(\\cdot\)for the Gaussian with the same mean and covariance as its argument,qa​bq\_\{ab\}minimizes

ℒportal\(qa​b\)=−𝔼qa​b\[12\(logqϕ\(a∣𝐱\)\+logqϕ\(b∣𝐱\)\)\]\+λDKL\(mm\(\(Ra​b\)\#qa​b\)∥qa​b\)\.\\mathcal\{L\}\_\{\\text\{portal\}\}\(q\_\{ab\}\)=\-\\mathbb\{E\}\_\{q\_\{ab\}\}\\\!\\left\[\\tfrac\{1\}\{2\}\\bigl\(\\log q\_\{\\phi\}\(a\\mid\\mathbf\{x\}\)\+\\log q\_\{\\phi\}\(b\\mid\\mathbf\{x\}\)\\bigr\)\\right\]\+\\lambda\\,D\_\{\\mathrm\{KL\}\}\\\!\\Bigl\(\\mathrm\{mm\}\\bigl\(\(R\_\{ab\}\)\_\{\\\#\}q\_\{ab\}\\bigr\)\\,\\Big\\\|\\,q\_\{ab\}\\Bigr\)\.\(9\)The first term is minimized at an exact tieqϕ​\(a∣𝐱\)=qϕ​\(b∣𝐱\)=12q\_\{\\phi\}\(a\\mid\\mathbf\{x\}\)=q\_\{\\phi\}\(b\\mid\\mathbf\{x\}\)=\\tfrac\{1\}\{2\}, since the two probabilities sum to at most one, and on its own already locates the overlap\. The second term favours the part of the tie set on which the round trip preservesqa​bq\_\{ab\}as a distribution\. This on its own does not enforce pointwise invertibility ofψa​b\\psi\_\{ab\}, but the defect can be repaired by an optional post\-processing step proposed in[Appendix C](https://arxiv.org/html/2610.33577#A3)\. Finally, expectations are estimated by reparameterized Monte Carlo, and each fit is initialized at the data point where the router most nearly ties between the two charts\. This differs from[Stolberg\-Larsen and Sommer \(2022\)](https://arxiv.org/html/2610.33577#bib.bib6), who cross charts directly at individual data points whose routing exceeds a threshold in both charts, so that the correspondence between charts exists only at those points\. A portal instead is a region of the overlap, and the transition map of[Eq\. 8](https://arxiv.org/html/2610.33577#S3.E8)lets charts be crossed from anywhere within it\.

![Refer to caption](https://arxiv.org/html/2610.33577v1/fig_cross_chart.png)Figure 4:An atlas of chart autoencoders can create smooth latent interpolation within each chart as well as across multiple charts\. After training, we precompute a transition graph \(right\)\. Interpolation between any two examples in any two charts can then be achieved by following the shortest path in that graph\. Within each chart’s latent space, we interpolate linearly \(left\)\.
#### Transition measure and chart graph\.

To interpolate between points in different charts, we need to know which charts are adjacent\. We use the model’s own generative process to determine this\. A sample drawn from chartaais routed byqϕq\_\{\\phi\}to some chartbbwith some probability which defines the Markov kernel

P⁡\(b∣a\)=∫qϕ​\(b∣ga​\(𝐳\)\)​p~​\(𝐳∣a\)​𝑑𝐳\.P\(b\\mid a\)=\\int q\_\{\\phi\}\\bigl\(b\\mid g\_\{a\}\(\\mathbf\{z\}\)\\bigr\)\\ \\tilde\{p\}\(\\mathbf\{z\}\\mid a\)\\ d\\mathbf\{z\}\.\(10\)We estimate this probability with Monte Carlo\. The kernel is row\-stochastic, and its diagonal is the probability of staying in chartaa\. Moreover, it is generally asymmetric, sinceP⁡\(b∣a\)P\(b\\mid a\)andP⁡\(a∣b\)P\(a\\mid b\)involve different decoders\. The charts and the off\-diagonal entries ofPPform a directed graph, the chart graph, with an edgea→ba\\to bof weightP⁡\(b∣a\)P\(b\\mid a\)\([Figure 4](https://arxiv.org/html/2610.33577#S3.F4), right\)\. Its integrand is the same routing probability whose tie defines the portalqa​bq\_\{ab\}in[Eq\. 9](https://arxiv.org/html/2610.33577#S3.E9), so every edge of the graph comes with a portal through which it can be traversed\.

#### Interpolation\.

With the transition kernel \([Eq\. 10](https://arxiv.org/html/2610.33577#S3.E10)\) precomputed, interpolating between points in chartsaaandbbreduces to a shortest\-path problem on the chart graph with edge costs−log⁡P⁡\(b∣a\)\-\\log P\(b\\mid a\)\. Since costs are additive, a chart sequencea=c0→c1→⋯→cK=ba=c\_\{0\}\\to c\_\{1\}\\to\\cdots\\to c\_\{K\}=bhas cost−∑klogP\(ck\+1∣ck\)\-\\sum\_\{k\}\\log P\(c\_\{k\+1\}\\mid c\_\{k\}\), which corresponds to the negative log\-probability of that sequence under the Markov chain defined byPP\. The shortest path, found with Dijkstra’s algorithm, is therefore the most probable route fromaatobb\. It passes through zero or more intermediate charts, entering and leaving each through a portal\. Within a chart we interpolate linearly to the portal of the next edge, represented by its mean𝐦ck​ck\+1\\mathbf\{m\}\_\{c\_\{k\}c\_\{k\+1\}\}or a sample drawn fromqck​ck\+1q\_\{c\_\{k\}c\_\{k\+1\}\}, and transfer through the transition map of[Eq\. 8](https://arxiv.org/html/2610.33577#S3.E8)\.

## 4Results

We train our model on three image datasets and compare it with a standard Variational Auto\-Encoder \(VAE\) and VaDE\([Jiang et al\., 2017](https://arxiv.org/html/2610.33577#bib.bib17)\)\. We also attempted to compare it with CAE\([Schonsheck et al\., 2019](https://arxiv.org/html/2610.33577#bib.bib13)\)and MLMM\-VAE\([Alberti et al\., 2024](https://arxiv.org/html/2610.33577#bib.bib20)\), but did not succeed in fitting these models to the datasets: one or two charts covered the entire dataset, while the remaining charts were inactive\.

All methods are scaled to use a similar number of parameters \(model size\), and hyperparameters are selected for each method and each dataset via grid search\. See[Appendix D](https://arxiv.org/html/2610.33577#A4)for a detailed description of the experimental setup\. Additional results are available in[Appendix F](https://arxiv.org/html/2610.33577#A6), including ablations, reconstructions and samples\. Throughout this paper, 2D visualizations of higher\-dimensional latent spaces use the first two PCA directions\.

#### Geometric examples\.

In a geometric setting, such as the double torus shown in[Figure 1](https://arxiv.org/html/2610.33577#S1.F1), our charts cover the surface with low distortion and clear separation\. Interpolation trajectories follow the data manifold closely, as shown in[Figure 8](https://arxiv.org/html/2610.33577#A6.F8)\.

#### Reconstruction & generation\.

Our method faithfully reconstructs input examples, as shown in[Figure 7](https://arxiv.org/html/2610.33577#A6.F7)\. Reconstruction performance for our method and the capacity\-matched baselines is also reported in[Table 1](https://arxiv.org/html/2610.33577#S4.T1)\. For reconstruction alone, the vanilla VAE tends to perform best, which can be attributed to its larger decoder capacity in comparison to a single chart\. This tendency is reversed in image generation, where our method outperforms the two baselines\. This can be explained by the better latent structure and the lower incidence of samples drawn from low\-density regions in the latent space \(see[Figure 5](https://arxiv.org/html/2610.33577#S4.F5)\)\. We can see that each chart’s latent space is closer to a Gaussian than the combined latent space of the VAE\. As a result, the generated samples have fewer artifacts\. More examples generated with our model can be found in[Figure 1](https://arxiv.org/html/2610.33577#S1.F1)and[Figure 9](https://arxiv.org/html/2610.33577#A6.F9)in the appendix\.

![Refer to caption](https://arxiv.org/html/2610.33577v1/fig_self_supervision_fashion_mnist.png)Figure 5:Sampling a model trained on Fashion\-MNIST\([Xiao et al\., 2017](https://arxiv.org/html/2610.33577#bib.bib12)\)along the two dominant latent PCA axes generates fewer degenerate samples than a parameter\-matched vanilla VAE\([Kingma and Welling, 2014](https://arxiv.org/html/2610.33577#bib.bib7)\)\. The charts also specialize without supervision, e\.g\. by placing bags without handles in a separate chart\. For readability, we show 16 out of the 20 charts\.![Refer to caption](https://arxiv.org/html/2610.33577v1/mnist_interpolation.png)\(a\)MNIST \(10 charts\)
![Refer to caption](https://arxiv.org/html/2610.33577v1/celeba_interpolation.png)\(b\)CelebA \(10 charts\)
![Refer to caption](https://arxiv.org/html/2610.33577v1/fashion_interpolation_20chart.png)\(c\)Fashion\-MNIST \(20 charts\)

Figure 6:Interpolations between sequences of randomly selected training samples \(green\)\. Our method may walk through one or more charts via portals \([Sec\. 3\.3](https://arxiv.org/html/2610.33577#S3.SS3)\), denoted by orange vertical bars\. The vanilla VAE traverses its single global latent space in a straight line, which leads to several nonsensical generations along the way\. In contrast, our method consistently stays on the data manifold, even if it entails covering more distance in latent space\.Table 1:Sampling and reconstruction quality for a plain VAE\([Kingma and Welling, 2014](https://arxiv.org/html/2610.33577#bib.bib7)\), VaDE\([Jiang et al\., 2017](https://arxiv.org/html/2610.33577#bib.bib17)\), and our atlas model \(10 charts\), all using the same number of parameters and training iterations\. Quality is measured by the Fréchet Inception Distance \(Inception\-v3,[Szegedy et al\. \(2016\)](https://arxiv.org/html/2610.33577#bib.bib9)\), averaged over three runs \(mean±\\pmstd\)\. Lower is better\.Sample FIDReconstruction FIDDatasetPlain VAEVaDEVAA \(ours\)Plain VAEVaDEVAA \(ours\)MNIST19\.84±\\pm0\.3414\.95±\\pm0\.5413\.74±\\pm0\.5516\.81±\\pm0\.2320\.54±\\pm0\.1218\.25±\\pm0\.13Fashion83\.27±\\pm1\.6245\.07±\\pm1\.6237\.73±\\pm1\.1231\.27±\\pm0\.3243\.49±\\pm0\.1532\.83±\\pm0\.38CelebA30\.43±\\pm0\.5928\.01±\\pm0\.2627\.84±\\pm0\.4216\.66±\\pm0\.1822\.09±\\pm0\.0717\.24±\\pm0\.25
#### Self\-supervised structure discovery\.

Our variational chart selection objective uncovers dataset structure by regularizing reconstruction smoothness while allowing disconnected charts when needed\. Without supervision, our 20\-chart VAA uncovers subclasses in Fashion\-MNIST and other datasets \(e\.g\., sandals and shoes with heels, see[Figure 5](https://arxiv.org/html/2610.33577#S4.F5)\)\. Moreover, we observe that the data closely follow the Gaussian prior in the latent domain, whereas this assumption breaks down in the capacity\-matched VAE\.

#### Interpolation\.

Clustering is not the only structure our method discovers\. The transition kernel \([Eq\. 10](https://arxiv.org/html/2610.33577#S3.E10)\) connects the charts into a graph that reflects semantic similarity without any supervision\. On MNIST \([Figure 4](https://arxiv.org/html/2610.33577#S3.F4)\), charts of similar digits such as 4 and 9 are strongly connected, while dissimilar ones such as 2 and 0 are nearly disconnected\. Interpolating along this graph \(see[Figure 4](https://arxiv.org/html/2610.33577#S3.F4),[Figure 6](https://arxiv.org/html/2610.33577#S4.F6)\) yields realistic samples along the entire path, including the passage through intermediate charts\. In contrast, the capacity\-matched VAE produces degenerate samples whenever its straight\-line path crosses a region of low data density\. The Variational Atlas Autoencoder avoids such regions because every segment of its path stays within a chart’s latent support and hence close to the data manifold\.

## 5Conclusion

We have introduced an end\-to\-end probabilistic method to train variational atlas autoencoders, which are inspired by the manifold hypothesis\. We proposed latent\-evidence\-based routing, which connects the probability of each data point under each chart’s prior to the confidence that the chart should encode it\. Finally, we showed how the mixture of charts can be turned into an atlas with a post\-training step, allowing cross\-chart interpolation trajectories that remain close to the data manifold\. Our method self\-supervises the input data, resulting in smooth charts while staying faithful to the data manifold, and enables high\-quality reconstruction and generation\. Due to our chart selection not all model parameters are active during sampling, resulting in a lower sampling cost while outperforming relevant baselines on sampling quality\.

In future work, determining an adequate chart count automatically, for example by adding or removing charts adaptively during training based on the data manifold’s complexity, would improve the method’s ease of use and utility\. Training can be further accelerated by training a lightweight routing proxy model, so that only the encoders in the most likely subset need to be evaluated, rather than all of them\. Next, because a single transition point is selected for each pair of charts, our cross\-chart interpolations sometimes cover more distance than needed\. Introducing more portals should help minimize total interpolation distance\. Finally, scaling the method to larger datasets will likely require combining it with more advanced architectures such as latent\-space diffusion or vector\-quantized VAEs\.

### AI use statement

In this work, we used large language models \(LLMs\) to prototype and develop code and to verify the concepts presented in this paper\. We have not used these tools to generate the core idea or the mathematics associated with it\. Additionally, we used LLM tools to help find and summarize related work, revise portions of the manuscript, and write code to build figures\. We have reviewed all AI\-assisted work for correctness\. In particular, LLM\-generated code was verified and tested by one author\. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI\.

### Ethics statement

While we do not foresee risks introduced by this method beyond those shared by all generative methods, foundational research can have a wide range of uses in subsequent work that translates it into practical applications\. We employed standard academic datasets and performed all experiments*in silico*\.

## References

- G\. S\. Alberti, J\. Hertrich, M\. Santacesaria, and S\. SciuttoManifold learning by mixture models of VAEs for inverse problems\.Journal of Machine Learning Research25\(202\),pp\. 1–35\.Cited by:[§D\.5](https://arxiv.org/html/2610.33577#A4.SS5.p3.1),[§1](https://arxiv.org/html/2610.33577#S1.p3.1),[§1](https://arxiv.org/html/2610.33577#S1.p4.1),[§2](https://arxiv.org/html/2610.33577#S2.SS0.SSS0.Px3.p2.1),[§3\.1](https://arxiv.org/html/2610.33577#S3.SS1.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2610.33577#S4.p1.1)\.
- Arvanitidiset al\.\(2018\)G\. Arvanitidis, L\. K\. Hansen, and S\. HaubergLatent space oddity: on the curvature of deep generative models\.In6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 \- May 3, 2018, Conference Track Proceedings,Cited by:[§1](https://arxiv.org/html/2610.33577#S1.p2.1)\.
- Arvanitidiset al\.\(2021\)G\. Arvanitidis, S\. Hauberg, and B\. SchölkopfGeometrically enriched latent spaces\.InThe 24th International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.130,pp\. 631–639\.Cited by:[§2](https://arxiv.org/html/2610.33577#S2.SS0.SSS0.Px2.p1.1)\.
- Batsonet al\.\(2021\)J\. Batson, C\. G\. Haaf, Y\. Kahn, and D\. A\. RobertsTopological obstructions to autoencoding\.Journal of High Energy Physics2021\(4\),pp\. 280\.Cited by:[§2](https://arxiv.org/html/2610.33577#S2.SS0.SSS0.Px1.p2.1)\.
- Brand \(2002\)M\. BrandCharting a manifold\.InAdvances in Neural Information Processing Systems,Vol\.15,pp\. 961–968\.Cited by:[§2](https://arxiv.org/html/2610.33577#S2.SS0.SSS0.Px3.p1.1)\.
- Brehmer and Cranmer \(2020\)J\. Brehmer and K\. CranmerFlows for simultaneous manifold learning and density estimation\.Advances in neural information processing systems33,pp\. 442–453\.Cited by:[§1](https://arxiv.org/html/2610.33577#S1.p1.1),[§2](https://arxiv.org/html/2610.33577#S2.SS0.SSS0.Px2.p1.1)\.
- Brownet al\.\(2023\)B\. C\. A\. Brown, A\. L\. Caterini, B\. L\. Ross, J\. C\. Cresswell, and G\. Loaiza\-GanemVerifying the union of manifolds hypothesis for image data\.InThe Eleventh International Conference on Learning Representations, ICLR,Cited by:[§1](https://arxiv.org/html/2610.33577#S1.p2.1),[§2](https://arxiv.org/html/2610.33577#S2.SS0.SSS0.Px1.p1.1)\.
- Feffermanet al\.\(2016\)C\. Fefferman, S\. Mitter, and H\. NarayananTesting the manifold hypothesis\.Journal of the American Mathematical Society29\(4\),pp\. 983–1049\.Cited by:[§A\.1](https://arxiv.org/html/2610.33577#A1.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.33577#S1.p2.1),[§2](https://arxiv.org/html/2610.33577#S2.SS0.SSS0.Px1.p1.1)\.
- Ghahramani and Hinton \(1996\)Z\. Ghahramani and G\. E\. HintonThe EM algorithm for mixtures of factor analyzers\.Technical reportTechnical ReportCRG\-TR\-96\-1,Department of Computer Science, University of Toronto\.Cited by:[§2](https://arxiv.org/html/2610.33577#S2.SS0.SSS0.Px3.p1.1)\.
- Hoffman and Johnson \(2016\)M\. D\. Hoffman and M\. J\. JohnsonELBO surgery: yet another way to carve up the variational evidence lower bound\.InWorkshop on Advances in Approximate Bayesian Inference, NeurIPS,Cited by:[§3\.2](https://arxiv.org/html/2610.33577#S3.SS2.SSS0.Px1.p1.1)\.
- Jianget al\.\(2017\)Z\. Jiang, Y\. Zheng, H\. Tan, B\. Tang, and H\. ZhouVariational deep embedding: an unsupervised and generative approach to clustering\.InProceedings of the 26th International Joint Conference on Artificial Intelligence,IJCAI’17,pp\. 1965–1972\.External Links:ISBN 9780999241103Cited by:[§2](https://arxiv.org/html/2610.33577#S2.SS0.SSS0.Px3.p2.1),[Table 1](https://arxiv.org/html/2610.33577#S4.T1),[§4](https://arxiv.org/html/2610.33577#S4.p1.1)\.
- Kingma and Welling \(2014\)D\. P\. Kingma and M\. WellingAuto\-encoding variational Bayes\.In2nd International Conference on Learning Representations, ICLR,Cited by:[§A\.3](https://arxiv.org/html/2610.33577#A1.SS3.SSS0.Px2.p1.1),[Appendix B](https://arxiv.org/html/2610.33577#A2.p6.1),[§3\.1](https://arxiv.org/html/2610.33577#S3.SS1.p1.3),[Figure 5](https://arxiv.org/html/2610.33577#S4.F5),[Table 1](https://arxiv.org/html/2610.33577#S4.T1)\.
- LeCunet al\.\(1998\)Y\. LeCun, L\. Bottou, Y\. Bengio, and P\. HaffnerGradient\-based learning applied to document recognition\.Proceedings of the IEEE86\(11\),pp\. 2278–2324\.Cited by:[Figure 1](https://arxiv.org/html/2610.33577#S1.F1)\.
- Lee \(2012\)J\. M\. LeeIntroduction to smooth manifolds\.2 edition,Graduate Texts in Mathematics, Vol\.218,Springer\.Cited by:[§A\.2](https://arxiv.org/html/2610.33577#A1.SS2.p2.1.1),[Appendix A](https://arxiv.org/html/2610.33577#A1.p1.1)\.
- Liuet al\.\(2015\)Z\. Liu, P\. Luo, X\. Wang, and X\. TangDeep learning face attributes in the wild\.InProceedings of International Conference on Computer Vision \(ICCV\),Cited by:[Figure 1](https://arxiv.org/html/2610.33577#S1.F1)\.
- Oringet al\.\(2021\)A\. Oring, Z\. Yakhini, and Y\. Hel\-OrAutoencoder image interpolation by shaping the latent space\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 8281–8290\.Cited by:[§1](https://arxiv.org/html/2610.33577#S1.p2.1)\.
- Radfordet al\.\(2016\)A\. Radford, L\. Metz, and S\. ChintalaUnsupervised representation learning with deep convolutional generative adversarial networks\.In4th International Conference on Learning Representations, ICLR,Cited by:[§D\.1](https://arxiv.org/html/2610.33577#A4.SS1.SSS0.Px2.p1.1)\.
- Rombachet al\.\(2022\)R\. Rombach, A\. Blattmann, D\. Lorenz, P\. Esser, and B\. OmmerHigh\-resolution image synthesis with latent diffusion models\.In2022 IEEE/CVF conference on computer vision and pattern recognition \(CVPR\),pp\. 10674–10685\.Cited by:[§1](https://arxiv.org/html/2610.33577#S1.p1.1)\.
- Schonshecket al\.\(2019\)S\. Schonsheck, J\. Chen, and R\. LaiChart auto\-encoders for manifold structured data\.arXiv preprint arXiv:1912\.10094\.Cited by:[§A\.3](https://arxiv.org/html/2610.33577#A1.SS3.SSS0.Px3.p1.1),[§D\.5](https://arxiv.org/html/2610.33577#A4.SS5.p3.1),[§1](https://arxiv.org/html/2610.33577#S1.p3.1),[§2](https://arxiv.org/html/2610.33577#S2.SS0.SSS0.Px3.p3.1),[§4](https://arxiv.org/html/2610.33577#S4.p1.1)\.
- Stolberg\-Larsen and Sommer \(2022\)J\. Stolberg\-Larsen and S\. SommerAtlas generative models and geodesic interpolation\.Image and Vision Computing122,pp\. 104433\.Cited by:[§2](https://arxiv.org/html/2610.33577#S2.SS0.SSS0.Px3.p3.1),[§3\.3](https://arxiv.org/html/2610.33577#S3.SS3.SSS0.Px2.p1.2)\.
- Szegedyet al\.\(2016\)C\. Szegedy, V\. Vanhoucke, S\. Ioffe, J\. Shlens, and Z\. WojnaRethinking the inception architecture for computer vision\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 2818–2826\.Cited by:[Table 1](https://arxiv.org/html/2610.33577#S4.T1)\.
- Tanielianet al\.\(2020\)U\. Tanielian, T\. Issenhuth, E\. Dohmatob, and J\. MaryLearning disconnected manifolds: a no GAN’s land\.InInternational Conference on Machine Learning,pp\. 9418–9427\.Cited by:[§2](https://arxiv.org/html/2610.33577#S2.SS0.SSS0.Px1.p2.1)\.
- Teh and Roweis \(2002\)Y\. W\. Teh and S\. T\. RoweisAutomatic alignment of local representations\.InAdvances in Neural Information Processing Systems,pp\. 841–848\.Cited by:[§2](https://arxiv.org/html/2610.33577#S2.SS0.SSS0.Px3.p1.1)\.
- Tipping and Bishop \(1999\)M\. E\. Tipping and C\. M\. BishopMixtures of probabilistic principal component analyzers\.Neural Computation11\(2\),pp\. 443–482\.Cited by:[§2](https://arxiv.org/html/2610.33577#S2.SS0.SSS0.Px3.p1.1)\.
- Vahdatet al\.\(2021\)A\. Vahdat, K\. Kreis, and J\. KautzScore\-based generative modeling in latent space\.Advances in Neural Information Processing Systems34,pp\. 11287–11302\.Cited by:[§1](https://arxiv.org/html/2610.33577#S1.p1.1)\.
- Xiaoet al\.\(2017\)H\. Xiao, K\. Rasul, and R\. VollgrafFashion\-MNIST: a novel image dataset for benchmarking machine learning algorithms\.arXiv preprint arXiv:1708\.07747\.External Links:cs\.LG/1708\.07747Cited by:[Figure 1](https://arxiv.org/html/2610.33577#S1.F1),[Figure 5](https://arxiv.org/html/2610.33577#S4.F5)\.
- Ye and Bors \(2022\)F\. Ye and A\. G\. BorsDeep mixture generative autoencoders\.IEEE Transactions on Neural Networks and Learning Systems33\(10\),pp\. 5789–5803\.Cited by:[§1](https://arxiv.org/html/2610.33577#S1.p3.1),[§2](https://arxiv.org/html/2610.33577#S2.SS0.SSS0.Px3.p2.1)\.
- Zhanget al\.\(2017\)D\. Zhang, Y\. Sun, B\. Eriksson, and L\. BalzanoDeep unsupervised clustering using mixture of autoencoders\.arXiv preprint arXiv:1712\.07788\.Cited by:[§D\.2](https://arxiv.org/html/2610.33577#A4.SS2.p2.1)\.
- Zhanget al\.\(2018\)R\. Zhang, P\. Isola, A\. A\. Efros, E\. Shechtman, and O\. WangThe unreasonable effectiveness of deep features as a perceptual metric\.In2018 IEEE/CVF conference on computer vision and pattern recognition,pp\. 586–595\.Cited by:[§D\.3](https://arxiv.org/html/2610.33577#A4.SS3.p2.1)\.

Appendices

## Appendix APreliminaries

We collect the geometric notions used in the main text and fix notation\. Everything in[Sec\. A\.1](https://arxiv.org/html/2610.33577#A1.SS1)is standard; we follow the conventions of[Lee \(2012\)](https://arxiv.org/html/2610.33577#bib.bib18)\.

### A\.1Manifolds, charts, and atlases

#### Manifolds\.

A*kk\-dimensional manifold*is a spaceℳ\\mathcal\{M\}that is locally Euclidean: every point has an open neighborhood homeomorphic to an open subset ofℝk\\mathbb\{R\}^\{k\}\. Throughout, we assume that the*manifold hypothesis*\([Fefferman et al\., 2016](https://arxiv.org/html/2610.33577#bib.bib19)\)holds: the data lie on \(or close to\) a manifoldℳ⊂ℝd\\mathcal\{M\}\\subset\\mathbb\{R\}^\{d\}withk≪dk\\ll d\.

#### Charts\.

A*chart*is a pair\(U,φ\)\(U,\\varphi\)whereU⊂ℳU\\subset\\mathcal\{M\}is open andφ:U→ℝk\\varphi\\colon U\\to\\mathbb\{R\}^\{k\}is a homeomorphism onto an open subset ofℝk\\mathbb\{R\}^\{k\}\. The mapφ\\varphiassigns local coordinates to points ofUU, and its inverseφ−1\\varphi^\{\-1\}parametrizesUU\. In autoencoder terms, the encoder plays the role ofφ\\varphiand the decoder that ofφ−1\\varphi^\{\-1\}\.

#### Atlases and transition maps\.

An*atlas*is a collection of charts\{\(Uc,φc\)\}c=1C\\\{\(U\_\{c\},\\varphi\_\{c\}\)\\\}\_\{c=1\}^\{C\}whose domains coverℳ\\mathcal\{M\}\. Where two charts overlap, their coordinates are related by the*transition map*

ψa​b=φb∘φa−1,\\psi\_\{ab\}=\\varphi\_\{b\}\\circ\\varphi\_\{a\}^\{\-1\},\(11\)defined onφa​\(Ua∩Ub\)\\varphi\_\{a\}\(U\_\{a\}\\cap U\_\{b\}\)\. Transition maps are themselves homeomorphisms, satisfyψb​a=ψa​b−1\\psi\_\{ba\}=\\psi\_\{ab\}^\{\-1\}, and compose consistently on triple overlaps:ψb​c∘ψa​b=ψa​c\\psi\_\{bc\}\\circ\\psi\_\{ab\}=\\psi\_\{ac\}\.[Sec\. 3\.3](https://arxiv.org/html/2610.33577#S3.SS3)measures how well the learned charts satisfy these conditions\.

#### Ball charts\.

The charts learned in this paper are slightly more restrictive than the general definition\. Each decoder is defined on all ofℝL\\mathbb\{R\}^\{L\}and the latent prior𝒩⁡\(𝟎,𝐈\)\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\)has full support, so each chart domain is \(approximately\) a deformed copy ofℝL\\mathbb\{R\}^\{L\}rather than an arbitrary open set\. We refer to such charts as*ball charts*\.

### A\.2Why a single chart is not enough

A standard VAE is a single ball chart: one decoder maps a connected latent spaceℝL\\mathbb\{R\}^\{L\}onto the data\. This can only faithfully representℳ\\mathcal\{M\}ifℳ\\mathcal\{M\}itself looks likeℝL\\mathbb\{R\}^\{L\}, which excludes most manifolds of interest\.

###### Proposition 1\.

Letg:ℝL→ℝdg\\colon\\mathbb\{R\}^\{L\}\\to\\mathbb\{R\}^\{d\}be continuous and letℳ⊂ℝd\\mathcal\{M\}\\subset\\mathbb\{R\}^\{d\}be compact or disconnected\. Thenggis not a homeomorphism ontoℳ\\mathcal\{M\}\. In particular,ℳ\\mathcal\{M\}cannot be covered by a single ball chart\.

###### Proof\.

Homeomorphisms preserve compactness and connectedness\([Lee, 2012](https://arxiv.org/html/2610.33577#bib.bib18)\)\. SinceℝL\\mathbb\{R\}^\{L\}is connected and not compact, so is any homeomorphic image of it\. ∎

Both hypotheses are typical of real data\. Bounded data such as images lie on compact sets, and multiclass data such as MNIST lie on several disconnected pieces\. In either case, at least two charts are needed\. The sphere, for example, requires exactly two\. The proposition also explains the spurious interpolants of single\-latent models discussed in[Sec\. 1](https://arxiv.org/html/2610.33577#S1)\. The imageg⁡\(ℝL\)g\(\\mathbb\{R\}^\{L\}\)of a single decoder is connected, so if it covers a disconnectedℳ\\mathcal\{M\}it must also pass through points outsideℳ\\mathcal\{M\}\. Off\-manifold decoding along latent paths between components is therefore unavoidable, not a failure of optimization\.

### A\.3Autoencoders as approximate charts

#### Autoencoders\.

An autoencoder consists of an encoderf:ℝd→ℝLf\\colon\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{L\}and a decoderg:ℝL→ℝdg\\colon\\mathbb\{R\}^\{L\}\\to\\mathbb\{R\}^\{d\}trained to minimize the reconstruction error𝔼p⁡\(𝐱\)​‖𝐱−g⁡\(f⁡\(𝐱\)\)‖2\\mathbb\{E\}\_\{p\(\\mathbf\{x\}\)\}\\\|\\mathbf\{x\}\-g\(f\(\\mathbf\{x\}\)\)\\\|^\{2\}\. With perfect reconstruction,\(f,g\)\(f,g\)would be a chart withU=ℳU=\\mathcal\{M\}\. Reconstruction is never perfect, so we call\(f,g\)\(f,g\)an*approximate chart*: a chart up to the reconstruction error\. The atlas properties studied in[Sec\. 3\.3](https://arxiv.org/html/2610.33577#S3.SS3)hold to the same accuracy\.

#### Variational autoencoders\.

A VAE\([Kingma and Welling, 2014](https://arxiv.org/html/2610.33577#bib.bib7)\)places a priorp⁡\(𝐳\)=𝒩⁡\(𝟎,𝐈\)p\(\\mathbf\{z\}\)=\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\)on the latent, modelspθ​\(𝐱∣𝐳\)p\_\{\\theta\}\(\\mathbf\{x\}\\mid\\mathbf\{z\}\)with the decoder, and approximates the posterior byqϕ​\(𝐳∣𝐱\)q\_\{\\phi\}\(\\mathbf\{z\}\\mid\\mathbf\{x\}\)using the encoder\. It is trained by maximizing the evidence lower bound

logpθ\(𝐱\)≥𝔼qϕ​\(𝐳∣𝐱\)\[logpθ\(𝐱∣𝐳\)\]−DKL\(qϕ\(𝐳∣𝐱\)∥p\(𝐳\)\),\\log p\_\{\\theta\}\(\\mathbf\{x\}\)\\geq\\mathbb\{E\}\_\{q\_\{\\phi\}\(\\mathbf\{z\}\\mid\\mathbf\{x\}\)\}\[\\log p\_\{\\theta\}\(\\mathbf\{x\}\\mid\\mathbf\{z\}\)\]\-D\_\{\\mathrm\{KL\}\}\\big\(q\_\{\\phi\}\(\\mathbf\{z\}\\mid\\mathbf\{x\}\)\\,\\\|\\,p\(\\mathbf\{z\}\)\\big\),\(12\)which is the single\-chart caseC=1C=1of[Eq\. 4](https://arxiv.org/html/2610.33577#S3.E4)\.

#### Chart autoencoders\.

Chart autoencoders\([Schonsheck et al\., 2019](https://arxiv.org/html/2610.33577#bib.bib13)\)useCCencoder–decoder pairs, one per chart\. Since a plain autoencoder has no likelihood, they require a separate network and loss to decide which chart a point belongs to\. In our model this decision follows from the ELBO itself \([Sec\. 3](https://arxiv.org/html/2610.33577#S3)\)\.

### A\.4Notation

For a mapffand a distributionμ\\mu,f\#​μf\_\{\\\#\}\\mudenotes the pushforward ofμ\\muunderff, i\.e\. the distribution off⁡\(𝐮\)f\(\\mathbf\{u\}\)for𝐮∼μ\\mathbf\{u\}\\sim\\mu\. For a distributionν\\nuonℝL\\mathbb\{R\}^\{L\},mm⁡\(ν\)\\mathrm\{mm\}\(\\nu\)denotes the Gaussian with the same mean and covariance\. A set of non\-negative functions\{wc\}\\\{w\_\{c\}\\\}with∑cwc​\(𝐱\)=1\\sum\_\{c\}w\_\{c\}\(\\mathbf\{x\}\)=1for every𝐱\\mathbf\{x\}is a*partition of unity*; the routing posteriorqϕ​\(c∣𝐱\)q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)is one by construction\.

## Appendix BDetailed derivation of the objective function

We provide the detailed derivation of the objective function\. To modelpθ​\(𝐱\)p\_\{\\theta\}\(\\mathbf\{x\}\)using multiple charts, we introduce a discrete chart variablec∈\{1,…,C\}c\\in\\\{1,\\ldots,C\\\}\. As a modeling assumption, each chart is assigned the same prior probability, so thatp⁡\(c\)=1/Cp\(c\)=1/C\. Each chart is further equipped with a continuousLL\-dimensional latent variable𝐳c∈ℝL\\mathbf\{z\}\_\{c\}\\in\\mathbb\{R\}^\{L\}with conditional priorp⁡\(𝐳c∣c\)=𝒩⁡\(𝟎,𝐈\)p\(\\mathbf\{z\}\_\{c\}\\mid c\)=\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\)\. Repeating[Eq\. 1](https://arxiv.org/html/2610.33577#S3.E1)for convenience, the resulting modeled data distribution is then given by:

pθ​\(𝐱\)=∑c=1Cp⁡\(c\)​∫pθ​\(𝐱∣𝐳c,c\)​p​\(𝐳c∣c\)​d​𝐳c\.p\_\{\\theta\}\(\\mathbf\{x\}\)=\\sum\_\{c=1\}^\{C\}p\(c\)\\int p\_\{\\theta\}\(\\mathbf\{x\}\\mid\\mathbf\{z\}\_\{c\},c\)\\,p\(\\mathbf\{z\}\_\{c\}\\mid c\)\\,\\mathrm\{d\}\\mathbf\{z\}\_\{c\}\.\(13\)
Given an observation𝐱\\mathbf\{x\}, inference requires the posterior distribution over the chart assignmentpθ​\(c∣𝐱\)p\_\{\\theta\}\(c\\mid\\mathbf\{x\}\)and the chart\-specific latent posteriorpθ​\(𝐳c∣c,𝐱\)p\_\{\\theta\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\. Since these posterior distributions are generally intractable, we introduce the variational approximationsqϕ​\(c∣𝐱\)q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)andqϕ​\(𝐳c∣c,𝐱\)q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\. Together, they define the approximate posterior \(repeating[Eq\. 2](https://arxiv.org/html/2610.33577#S3.E2)for convenience\)

qϕ​\(c,𝐳c∣𝐱\)=qϕ​\(c∣𝐱\)​qϕ​\(𝐳c∣c,𝐱\),q\_\{\\phi\}\(c,\\mathbf\{z\}\_\{c\}\\mid\\mathbf\{x\}\)=q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\\,q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\),\(14\)
which approximates the true posteriorpθ​\(c,𝐳c∣𝐱\)p\_\{\\theta\}\(c,\\mathbf\{z\}\_\{c\}\\mid\\mathbf\{x\}\)\.

Direct evaluation and optimization of the marginal log\-likelihood in[Eq\. 13](https://arxiv.org/html/2610.33577#A2.E13)are generally intractable because the integral over the continuous latent variables cannot usually be computed in closed form\. As in the single\-chart VAE setting\([Kingma and Welling, 2014](https://arxiv.org/html/2610.33577#bib.bib7)\), we therefore optimize a variational lower bound onlog⁡pθ​\(𝐱\)\\log p\_\{\\theta\}\(\\mathbf\{x\}\)\.

Usingqϕ​\(c,𝐳c∣𝐱\)=qϕ​\(c∣𝐱\)​qϕ​\(𝐳c∣c,𝐱\)q\_\{\\phi\}\(c,\\mathbf\{z\}\_\{c\}\\mid\\mathbf\{x\}\)=q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\), the marginal likelihood can be rewritten as

log⁡pθ​\(𝐱\)\\displaystyle\\log p\_\{\\theta\}\(\\mathbf\{x\}\)=log∑c=1C∫qϕ\(c,𝐳c∣𝐱\)pθ​\(𝐱,𝐳c,c\)qϕ​\(c,𝐳c∣𝐱\)d𝐳c\.\\displaystyle=\\log\\sum\_\{c=1\}^\{C\}\\int q\_\{\\phi\}\(c,\\mathbf\{z\}\_\{c\}\\mid\\mathbf\{x\}\)\\frac\{p\_\{\\theta\}\(\\mathbf\{x\},\\mathbf\{z\}\_\{c\},c\)\}\{q\_\{\\phi\}\(c,\\mathbf\{z\}\_\{c\}\\mid\\mathbf\{x\}\)\}\\,\\mathrm\{d\}\\mathbf\{z\}\_\{c\}\.\(15\)Applying Jensen’s inequality yields

log⁡pθ​\(𝐱\)\\displaystyle\\log p\_\{\\theta\}\(\\mathbf\{x\}\)≥∑c=1Cqϕ​\(c∣𝐱\)​∫qϕ​\(𝐳c∣c,𝐱\)​log⁡pθ​\(𝐱,𝐳c,c\)qϕ​\(c,𝐳c∣𝐱\)​d​𝐳c\\displaystyle\\geq\\sum\_\{c=1\}^\{C\}q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\\int q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\\log\\frac\{p\_\{\\theta\}\(\\mathbf\{x\},\\mathbf\{z\}\_\{c\},c\)\}\{q\_\{\\phi\}\(c,\\mathbf\{z\}\_\{c\}\\mid\\mathbf\{x\}\)\}\\,\\mathrm\{d\}\\mathbf\{z\}\_\{c\}=𝔼c∼qϕ​\(c∣𝐱\)​\[𝔼𝐳c∼qϕ​\(𝐳c∣c,𝐱\)​\[log⁡pθ​\(𝐱,𝐳c,c\)qϕ​\(c,𝐳c∣𝐱\)\]\]\.\\displaystyle=\\mathbb\{E\}\_\{c\\sim q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\}\\left\[\\mathbb\{E\}\_\{\\mathbf\{z\}\_\{c\}\\sim q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\}\\left\[\\log\\frac\{p\_\{\\theta\}\(\\mathbf\{x\},\\mathbf\{z\}\_\{c\},c\)\}\{q\_\{\\phi\}\(c,\\mathbf\{z\}\_\{c\}\\mid\\mathbf\{x\}\)\}\\right\]\\right\]\.This defines the evidence lower bound for the multi\-chart model,

log⁡pθ​\(𝐱\)≥ℒELBO​\(𝐱\)=𝔼qϕ​\(c,𝐳c∣𝐱\)​\[log⁡pθ​\(𝐱,𝐳c,c\)qϕ​\(c,𝐳c∣𝐱\)\]\.\\log p\_\{\\theta\}\(\\mathbf\{x\}\)\\geq\\mathcal\{L\}\_\{\\mathrm\{ELBO\}\}\(\\mathbf\{x\}\)=\\mathbb\{E\}\_\{q\_\{\\phi\}\(c,\\mathbf\{z\}\_\{c\}\\mid\\mathbf\{x\}\)\}\\left\[\\log\\frac\{p\_\{\\theta\}\(\\mathbf\{x\},\\mathbf\{z\}\_\{c\},c\)\}\{q\_\{\\phi\}\(c,\\mathbf\{z\}\_\{c\}\\mid\\mathbf\{x\}\)\}\\right\]\.The lower bound can equivalently be related to the exact posterior through

logpθ\(𝐱\)=ℒELBO\(𝐱\)\+DKL\(qϕ\(c,𝐳c∣𝐱\)∥pθ\(c,𝐳c∣𝐱\)\)\.\\log p\_\{\\theta\}\(\\mathbf\{x\}\)=\\mathcal\{L\}\_\{\\mathrm\{ELBO\}\}\(\\mathbf\{x\}\)\+D\_\{\\mathrm\{KL\}\}\\\!\\left\(q\_\{\\phi\}\(c,\\mathbf\{z\}\_\{c\}\\mid\\mathbf\{x\}\)\\,\\\|\\,p\_\{\\theta\}\(c,\\mathbf\{z\}\_\{c\}\\mid\\mathbf\{x\}\)\\right\)\.Since the KL divergence is non\-negative, it follows thatlog⁡pθ​\(𝐱\)≥ℒELBO​\(𝐱\)\\log p\_\{\\theta\}\(\\mathbf\{x\}\)\\geq\\mathcal\{L\}\_\{\\mathrm\{ELBO\}\}\(\\mathbf\{x\}\)\. Using the factorizationpθ​\(𝐱,𝐳c,c\)=pθ​\(𝐱∣𝐳c,c\)​p​\(𝐳c∣c\)​p​\(c\)p\_\{\\theta\}\(\\mathbf\{x\},\\mathbf\{z\}\_\{c\},c\)=p\_\{\\theta\}\(\\mathbf\{x\}\\mid\\mathbf\{z\}\_\{c\},c\)\\,p\(\\mathbf\{z\}\_\{c\}\\mid c\)\\,p\(c\)andqϕ​\(c,𝐳c∣𝐱\)=qϕ​\(𝐳c∣c,𝐱\)​qϕ​\(c∣𝐱\)q\_\{\\phi\}\(c,\\mathbf\{z\}\_\{c\}\\mid\\mathbf\{x\}\)=q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\\,q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\), the ELBO can be written as

ℒELBO​\(𝐱\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{ELBO\}\}\(\\mathbf\{x\}\)=𝔼c∼qϕ​\(c∣𝐱\)​𝔼𝐳c∼qϕ​\(𝐳c∣c,𝐱\)​\[log⁡pθ​\(𝐱∣𝐳c,c\)​p​\(𝐳c∣c\)​p​\(c\)qϕ​\(𝐳c∣c,𝐱\)​qϕ​\(c∣𝐱\)\]\\displaystyle=\\mathbb\{E\}\_\{c\\sim q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\}\\mathbb\{E\}\_\{\\mathbf\{z\}\_\{c\}\\sim q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\}\\left\[\\log\\frac\{p\_\{\\theta\}\(\\mathbf\{x\}\\mid\\mathbf\{z\}\_\{c\},c\)\\,p\(\\mathbf\{z\}\_\{c\}\\mid c\)\\,p\(c\)\}\{q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\\,q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\}\\right\]=𝔼c∼qϕ​\(c∣𝐱\)​𝔼𝐳c∼qϕ​\(𝐳c∣c,𝐱\)​\[log⁡pθ​\(𝐱∣𝐳c,c\)\]\\displaystyle=\\mathbb\{E\}\_\{c\\sim q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\}\\mathbb\{E\}\_\{\\mathbf\{z\}\_\{c\}\\sim q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\}\\left\[\\log p\_\{\\theta\}\(\\mathbf\{x\}\\mid\\mathbf\{z\}\_\{c\},c\)\\right\]\+𝔼c∼qϕ​\(c∣𝐱\)​𝔼𝐳c∼qϕ​\(𝐳c∣c,𝐱\)​\[log⁡p⁡\(𝐳c∣c\)qϕ​\(𝐳c∣c,𝐱\)\]\\displaystyle\\quad\+\\mathbb\{E\}\_\{c\\sim q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\}\\mathbb\{E\}\_\{\\mathbf\{z\}\_\{c\}\\sim q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\}\\left\[\\log\\frac\{p\(\\mathbf\{z\}\_\{c\}\\mid c\)\}\{q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\}\\right\]\+𝔼c∼qϕ​\(c∣𝐱\)​\[log⁡p⁡\(c\)qϕ​\(c∣𝐱\)\]\.\\displaystyle\\quad\+\\mathbb\{E\}\_\{c\\sim q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\}\\left\[\\log\\frac\{p\(c\)\}\{q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\}\\right\]\.Expressing the last two terms as KL divergences gives

ℒELBO​\(𝐱\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{ELBO\}\}\(\\mathbf\{x\}\)=𝔼c∼qϕ​\(c∣𝐱\)​𝔼𝐳c∼qϕ​\(𝐳c∣c,𝐱\)​\[log⁡pθ​\(𝐱∣𝐳c,c\)\]\\displaystyle=\\mathbb\{E\}\_\{c\\sim q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\}\\mathbb\{E\}\_\{\\mathbf\{z\}\_\{c\}\\sim q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\}\\left\[\\log p\_\{\\theta\}\(\\mathbf\{x\}\\mid\\mathbf\{z\}\_\{c\},c\)\\right\]−𝔼c∼qϕ​\(c∣𝐱\)\[DKL\(qϕ\(𝐳c∣c,𝐱\)∥p\(𝐳c∣c\)\)\]\\displaystyle\\quad\-\\mathbb\{E\}\_\{c\\sim q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\}\\left\[D\_\{\\mathrm\{KL\}\}\\\!\\left\(q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\\,\\\|\\,p\(\\mathbf\{z\}\_\{c\}\\mid c\)\\right\)\\right\]−DKL\(qϕ\(c∣𝐱\)∥p\(c\)\)\.\\displaystyle\\quad\-D\_\{\\mathrm\{KL\}\}\\\!\\left\(q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\\,\\\|\\,p\(c\)\\right\)\.Writing the chart router as an explicit finite sum makes it clear that the per\-chart ELBO

ℒc\(𝐱\)=𝔼qϕ​\(𝐳c∣c,𝐱\)\[logpθ\(𝐱∣𝐳c,c\)\]−DKL\(qϕ\(𝐳c∣c,𝐱\)∥p\(𝐳c∣c\)\)\\mathcal\{L\}\_\{c\}\(\\mathbf\{x\}\)=\\mathbb\{E\}\_\{q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\}\\\!\\left\[\\log p\_\{\\theta\}\(\\mathbf\{x\}\\mid\\mathbf\{z\}\_\{c\},c\)\\right\]\-D\_\{\\mathrm\{KL\}\}\\\!\\left\(q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\\,\\\|\\,p\(\\mathbf\{z\}\_\{c\}\\mid c\)\\right\)\([5](https://arxiv.org/html/2610.33577#S3.E5)\)is simply weighted by each chart’s routing probabilityqϕ​\(c∣𝐱\)q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\), while the prior routing termDKL\(qϕ\(c∣𝐱\)∥p\(c\)\)D\_\{\\mathrm\{KL\}\}\(q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\\,\\\|\\,p\(c\)\)regularizes the routing function, yielding[Eq\. 4](https://arxiv.org/html/2610.33577#S3.E4)\. In practice, we replace the log\-likelihood by a reconstruction lossℒrecon​\(𝐱,gc​\(𝐳c\)\)\\mathcal\{L\}\_\{\\mathrm\{recon\}\}\(\\mathbf\{x\},g\_\{c\}\(\\mathbf\{z\}\_\{c\}\)\), which defaults to−log⁡pθ​\(𝐱∣𝐳c,c\)\-\\log p\_\{\\theta\}\(\\mathbf\{x\}\\mid\\mathbf\{z\}\_\{c\},c\)but may also be a perceptual loss \([Appendix D](https://arxiv.org/html/2610.33577#A4)\), and weight the regularizers by the hyperparametersβ\\beta\(KL weight\)\. This yields the per\-chart loss

ℓc\(𝐱\)=𝔼qϕ​\(𝐳c∣c,𝐱\)\[ℒrecon\(𝐱,gc\(𝐳c\)\)\]\+βDKL\(qϕ\(𝐳c∣c,𝐱\)∥p\(𝐳c∣c\)\)\\ell\_\{c\}\(\\mathbf\{x\}\)=\\mathbb\{E\}\_\{q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\}\\left\[\\mathcal\{L\}\_\{\\mathrm\{recon\}\}\\\!\\left\(\\mathbf\{x\},g\_\{c\}\(\\mathbf\{z\}\_\{c\}\)\\right\)\\right\]\+\\beta\\,D\_\{\\mathrm\{KL\}\}\\\!\\left\(q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\\,\\\|\\,p\(\\mathbf\{z\}\_\{c\}\\mid c\)\\right\)\(16\)and the training loss

ℓ\(𝐱\)=DKL\(qϕ\(c∣𝐱\)∥p\(c\)\)\+∑c=1Cqϕ\(c∣𝐱\)ℓc\(𝐱\),\\ell\(\\mathbf\{x\}\)=D\_\{\\mathrm\{KL\}\}\\\!\\left\(q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\\,\\\|\\,p\(c\)\\right\)\+\\sum\_\{c=1\}^\{C\}q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\\,\\ell\_\{c\}\(\\mathbf\{x\}\),\(17\)whereqϕ​\(c∣𝐱\)q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)is the router of[Eq\. 7](https://arxiv.org/html/2610.33577#S3.E7)at temperatureTT\. The loss coincides with−ℒELBO\-\\mathcal\{L\}\_\{\\mathrm\{ELBO\}\}from[Eq\. 4](https://arxiv.org/html/2610.33577#S3.E4)whenβ=1\\beta=1andℒrecon=−log⁡pθ\\mathcal\{L\}\_\{\\mathrm\{recon\}\}=\-\\log p\_\{\\theta\}\.

## Appendix CReducing the roundtrip error

In the portal objective[Eq\. 9](https://arxiv.org/html/2610.33577#S3.E9), the second term preservesqa​bq\_\{ab\}as a distribution\. This locates the overlap but does not makeψa​b\\psi\_\{ab\}invertible on it\. To fix this, we propose an optional lightweight fine\-tuning step in which the expected pointwise defect,

ℒatlas=𝔼\(a,b\)∼π​\[𝔼𝐳∼qa​b​‖Ra​b​\(𝐳\)−𝐳‖a2\+𝔼𝐰∼\(ψa​b\)\#​qa​b​‖Rb​a​\(𝐰\)−𝐰‖b2\],\\mathcal\{L\}\_\{\\mathrm\{atlas\}\}=\\mathbb\{E\}\_\{\(a,b\)\\sim\\pi\}\\Bigl\[\\mathbb\{E\}\_\{\\mathbf\{z\}\\sim q\_\{ab\}\}\\\|R\_\{ab\}\(\\mathbf\{z\}\)\-\\mathbf\{z\}\\\|\_\{a\}^\{2\}\+\\mathbb\{E\}\_\{\\mathbf\{w\}\\sim\(\\psi\_\{ab\}\)\_\{\\\#\}q\_\{ab\}\}\\\|R\_\{ba\}\(\\mathbf\{w\}\)\-\\mathbf\{w\}\\\|\_\{b\}^\{2\}\\Bigr\],\(18\)is minimized in both directions, jointly with the training loss[Eq\. 17](https://arxiv.org/html/2610.33577#A2.E17)as a retention term\. Distances are measured in the Mahalanobis metric∥⋅∥c\\\|\\cdot\\\|\_\{c\}defined above, so that a defect of one is the size of a typical latent and defects are comparable across charts\. The outer expectation is estimated by drawing pairs fromπ\\piwith replacement and the inner expectations by reparameterized Monte Carlo with gradients throughψa​b\\psi\_\{ab\}\. The retention term ensures that the model retains its sampling and reconstruction performance\.

## Appendix DExperimental setup

This appendix provides additional details on the setup summarized in[Sec\. 4](https://arxiv.org/html/2610.33577#S4), specifically on the architecture \(Sec\.[D\.1](https://arxiv.org/html/2610.33577#A4.SS1)\), the chart\-balance term and its weightγ\\gamma\(Sec\.[D\.2](https://arxiv.org/html/2610.33577#A4.SS2)\), per\-dataset hyperparameters \(Secs\.[D\.3](https://arxiv.org/html/2610.33577#A4.SS3)\-[D\.4](https://arxiv.org/html/2610.33577#A4.SS4)\), and how the vanilla VAE and VaDE baselines are made comparable to ours \(Sec\.[D\.5](https://arxiv.org/html/2610.33577#A4.SS5)\)\.

### D\.1Network Architecture

We use two different network architectures: an MLP\-only architecture for the double torus dataset, and a convolutional architecture for the image\-based datasets\.

#### MLP \(double torus\)\.

The encoders and decoders aredepth\-layer,hidden\-wide dense networks with ReLU nonlinearities, where each encoder/decoder pair is private to its respective chart\. This can be seen asCCseparate pairs of dense MLPs, one per chart; although in practice we implement this as a single batched matrix multiply over allCCcharts at once, rather than looping overCCseparate layers, which is significantly faster on a GPU\. The encoder maps the 3D ambient coordinates that lie on the surface of the torus to a2×2\\timeslatent\_dimlatent vector containing the mean and log\-variance; the decoder mirrors this back to 3D, and its raw output is used directly as the reconstruction, with no output nonlinearity \(Sec\.[D\.4](https://arxiv.org/html/2610.33577#A4.SS4)\)\.

#### DCGAN \(MNIST, Fashion\-MNIST, CelebA\)\.

For image datasets, we use an architecture that can exploit their spatial structure more effectively\. The encoder and decoder follow the convolutional architecture of[Radford et al\. \(2016\)](https://arxiv.org/html/2610.33577#bib.bib8), with the channel width doubling at every downsampling stage and, conversely, halving at every upsampling stage\. In every tuned configuration reported in this paper, this entire convolutional network \(encoder and decoder\) is shared identically across all charts \(Table[2](https://arxiv.org/html/2610.33577#A4.T2)\)\. Charts only differ at the very end: the shared encoder’s final feature map is flattened and passed through one linear layer per chart, mapping it to that chart’s own2×2\\timeslatent\_dimlatent vector containing the mean and log\-variance; the decoder mirrors this with one linear layer per chart that expandszzback to that same feature\-map size, before the \(again shared\) network decodes it into an image\. These per\-chart linear layers are the only chart\-specific parameters in this backbone\. The VAE and VaDE baselines \(Sec\.[D\.5](https://arxiv.org/html/2610.33577#A4.SS5)\) reuse this same convolutional network atC=1C=1\.

The architecture also supports making some of the deepest convolutional stages chart\-specific as well, rather than only the two linear layers\. Reducing the size of the shared encoder/decoder model improves the generation speed, but comes with the trade\-off that each chart needs to learn the mapping from ambient to latent space from scratch, effectively re\-learning the feature extractor\.

Table 2:Resolved DCGAN dimensions per dataset: channel widths of the shared convolutional network and the size of the feature map passed to the per\-chart linear layers\.MNISTFashion\-MNISTCelebAConvolutional stages224Channel widths64, 12864, 12864, 128, 256, 512Bottleneck channels13651Bottleneck resolution7×77\\times 77×77\\times 74×44\\times 4

### D\.2The Balance Weightγ\\gamma

Although[Eq\. 4](https://arxiv.org/html/2610.33577#S3.E4)already contains a routing KL toward the uniform chart prior, we keep that term at weight11and do not tune it\. The hyperparameterγ\\gammainstead weights a separate*balance term*, which compares the uniform distributionUUover theCCcharts with the routing averaged over a minibatch,r¯c=1B​∑i=1Bqϕ​\(c∣xi\)\\bar\{r\}\_\{c\}=\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}q\_\{\\phi\}\(c\\mid x\_\{i\}\)\. The loss we minimize is therefore

ℒ=−ℒELBO\+γDKL\(U∥r¯\),\\mathcal\{L\}=\-\\mathcal\{L\}\_\{\\mathrm\{ELBO\}\}\+\\gamma\\,D\_\{\\mathrm\{KL\}\}\(U\\,\\\|\\,\\bar\{r\}\),\(19\)withγ=50\\gamma=50for the entire run in every configuration\.

The two terms push in different ways\. The routing KL of[Eq\. 4](https://arxiv.org/html/2610.33577#S3.E4)acts on each sample and pulls everyqϕ​\(c∣x\)q\_\{\\phi\}\(c\\mid x\)toward uniform, which also discourages confident chart assignments\. The balance term acts only on the batch average: individual points may still be routed sharply to a single chart, as long as all charts are used about equally overall\. To see why, note that the batch\-averaged routing KL splits as1B∑iDKL\(ri∥U\)=I^\(X;C\)\+DKL\(r¯∥U\)\\frac\{1\}\{B\}\\sum\_\{i\}D\_\{\\mathrm\{KL\}\}\(r\_\{i\}\\,\\\|\\,U\)=\\hat\{I\}\(X;C\)\+D\_\{\\mathrm\{KL\}\}\(\\bar\{r\}\\,\\\|\\,U\), withri=qϕ​\(c∣xi\)r\_\{i\}=q\_\{\\phi\}\(c\\mid x\_\{i\}\)andI^\(X;C\)=1B∑iDKL\(ri∥r¯\)\\hat\{I\}\(X;C\)=\\frac\{1\}\{B\}\\sum\_\{i\}D\_\{\\mathrm\{KL\}\}\(r\_\{i\}\\,\\\|\\,\\bar\{r\}\)the empirical mutual information between inputs and charts\. The first part penalizes confident routing, the second unequal chart usage\. Theγ\\gammaterm amplifies only the latter \(in reversed form\), so charts are kept alive without blurring the assignments\. This is the same role played by the auxiliary load\-balancing loss in related work\([Zhang et al\., 2017](https://arxiv.org/html/2610.33577#bib.bib1)\), which prevents a router from sending all inputs to a few expert charts\.

### D\.3Image Datasets

Hyperparameters \([Table 3](https://arxiv.org/html/2610.33577#A4.T3)\) are selected independently per dataset via grid search\. The VAE and VaDE baselines reuse the architecture of Sec\.[D\.1](https://arxiv.org/html/2610.33577#A4.SS1)and the reconstruction loss below, with only width rescaled to match parameter count \(Sec\.[D\.5](https://arxiv.org/html/2610.33577#A4.SS5)\)\.

Table 3:Tuned hyperparameters for MNIST, Fashion\-MNIST, and CelebA\.HyperparameterMNISTFashion\-MNISTCelebACharts102010Latent dimension1616128Reconstruction lossBernoulli \(BCE\)Bernoulli \(BCE\)L1 \+ LPIPS, scale×3\\times 3TT\(routing temperature\)Fixed at 1\.0Fixed at 1\.0Annealed6\.0→2\.06\.0\\to 2\.0β\\beta\(KL weight\)0\.50\.10\.1γ\\gamma\(load\-balancing weight\)50, entire run50, entire run50, entire runLearning rate5×10−45\\times 10^\{\-4\}2×10−32\\times 10^\{\-3\}5×10−45\\times 10^\{\-4\}LR scheduleCos\-annealed to 1%Cos\-annealed to 1%Cos\-annealed to 1%Batch size256256128Epochs200200200A per\-pixel likelihood scores each pixel independently, which rewards blurry reconstructions\. For CelebA specifically, we replace it with a mix of L1 error in pixel space and LPIPS\([Zhang et al\., 2018](https://arxiv.org/html/2610.33577#bib.bib5)\), a common perceptual loss used by latent\-diffusion autoencoders\. Rather than comparing raw pixels, LPIPS compares distances in a pretrained latent space that tracks human judgments more closely\. We weight the two terms so that each contributes about as many nats as the part of the BCE loss it replaces, split evenly, then scale the sum by 3 \(see[Table 3](https://arxiv.org/html/2610.33577#A4.T3)\)\.

Two optimizations keep this loss practical\. LPIPS is by far the most expensive term to compute, so we evaluate it once per image rather than once per chart: every chart still gets its own L1 term, but all charts share the LPIPS score of whichever chart reconstructs best\. This leaves the ranking between charts unchanged while avoiding the slowdown from scoring every chart separately\. We also run the LPIPS network at reduced precision, which roughly halves its cost with no measurable effect on training\.

### D\.4Double Torus

The double torus is a synthetic point cloud sampled uniformly by area from a genus\-2 surface embedded inℝ3\\mathbb\{R\}^\{3\}\(the surface of[Figure 1](https://arxiv.org/html/2610.33577#S1.F1)\)\. Targets are 3D coordinates rather than pixel intensities, so we use a squared\-error reconstruction term directly on the decoder’s raw output, rather than passing it through a sigmoid first as we do for pixel intensities\. This term is small relative to the KL term and induces posterior collapse if left uncorrected\. We counteract this by scaling up the reconstruction term \(200×200\\times\), with no other change to the loss, routing, or training procedure\. Table[4](https://arxiv.org/html/2610.33577#A4.T4)lists the full configuration\.

Table 4:Tuned hyperparameters for the double torus\.HyperparameterValueBackboneMLP, hidden width 512, depth 5Charts4Latent dimension2Reconstruction lossSquared error, no output squashing, scale×200\\times 200TT\(routing temperature\)annealed1\.0→0\.21\.0\\to 0\.2β\\beta\(KL weight\)1\.5γ\\gamma\(load\-balancing weight\)50, for the entire runLearning rate1×10−31\\times 10^\{\-3\}, cosine\-annealed to 1%Batch size2048Epochs100Surface points1,000,0001\{,\}000\{,\}000
### D\.5Baselines

Both baselines are given our model’s total trainable parameter count on each dataset: we perform a binary search over the baseline’s width parameter \(channel count for DCGAN, hidden width for the MLP\) to match that target, rather than altering depth or other architectural choices, and train for the same number of epochs with the same reconstruction loss\. The vanilla VAE control is simplyC=1C=1\(Sec\. A\.3\), so matching parameters amounts to retuning the backbone width atC=1C=1\.

For VaDE, we port the original authors’ code to modern PyTorch, verify it reproduces their reported numbers, and then swap its Gaussian\-mixture prior onto our own single\-chart DCGAN backbone \(Sec\.[D\.1](https://arxiv.org/html/2610.33577#A4.SS1)\) in place of their original autoencoder\. So the architecture, parameter count, and training length are identical across all three methods, and only the modeling choice \(charts vs\. GMM prior vs\. single Gaussian prior\) differs\.

As noted in[Sec\. 4](https://arxiv.org/html/2610.33577#S4), we also attempted CAE\([Schonsheck et al\., 2019](https://arxiv.org/html/2610.33577#bib.bib13)\)and MLMM\-VAE\([Alberti et al\., 2024](https://arxiv.org/html/2610.33577#bib.bib20)\)but did not succeed in fitting either to the image\-based datasets: one or two charts covered the whole dataset, while the remaining charts were effectively inactive\. As a result these methods were only using a small subset of the available model\-parameter capacity resulting in significantly higher FID when sampling new data from the model\. For that reason, we exclude them from the main comparison\.

## Appendix EAlgorithm

Algorithm 1Training procedure for Variational Atlas Autoencoders\. AllCCencoders are evaluated for each minibatch to obtain the routing weights and per\-chart losses; encoders and decoders are updated jointly\.1:data

\{𝐱\}\\\{\\mathbf\{x\}\\\}, encoder–decoder pairs

\{\(fc,gc\)\}c=1C\\\{\(f\_\{c\},g\_\{c\}\)\\\}\_\{c=1\}^\{C\}, temperature

TT, weights

β,γ\\beta,\\gamma
2:whilenot convergeddo

3:sample a minibatch

𝐱\\mathbf\{x\}
4:for

c=1,…,Cc=1,\\dots,Cdo

5:

\(𝝁c,𝚺c\)←fc​\(𝐱\)\(\\bm\{\\mu\}\_\{c\},\\bm\{\\Sigma\}\_\{c\}\)\\leftarrow f\_\{c\}\(\\mathbf\{x\}\)⊳\\trianglerightencode with every chart

6:

sc←log⁡𝒩⁡\(𝝁c,0,𝐈\+𝚺c\)s\_\{c\}\\leftarrow\\log\\mathcal\{N\}\(\\bm\{\\mu\}\_\{c\};\\,\\mathbf\{0\},\\,\\mathbf\{I\}\+\\bm\{\\Sigma\}\_\{c\}\)⊳\\trianglerightlatent evidence,[Eq\. 7](https://arxiv.org/html/2610.33577#S3.E7)

7:

𝐳c←𝝁c\+𝚺c1/2​ϵ\\mathbf\{z\}\_\{c\}\\leftarrow\\bm\{\\mu\}\_\{c\}\+\\bm\{\\Sigma\}\_\{c\}^\{1/2\}\\bm\{\\epsilon\},

ϵ∼𝒩⁡\(𝟎,𝐈\)\\bm\{\\epsilon\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\)⊳\\trianglerightreparameterized sample

8:

ℓc←ℒrecon\(𝐱,gc\(𝐳c\)\)\+βDKL\(qϕ\(𝐳c∣c,𝐱\)∥p\(𝐳c∣c\)\)\\ell\_\{c\}\\leftarrow\\mathcal\{L\}\_\{\\mathrm\{recon\}\}\(\\mathbf\{x\},g\_\{c\}\(\\mathbf\{z\}\_\{c\}\)\)\+\\beta\\,D\_\{\\mathrm\{KL\}\}\(q\_\{\\phi\}\(\\mathbf\{z\}\_\{c\}\\mid c,\\mathbf\{x\}\)\\,\\\|\\,p\(\\mathbf\{z\}\_\{c\}\\mid c\)\)⊳\\trianglerightper\-chart loss,[Eq\. 16](https://arxiv.org/html/2610.33577#A2.E16)

9:endfor

10:

qϕ​\(c∣𝐱\)←softmaxc⁡\(sc/T\+log⁡p⁡\(c\)\)q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\\leftarrow\\operatorname\{softmax\}\_\{c\}\\\!\\left\(s\_\{c\}/T\+\\log p\(c\)\\right\)⊳\\trianglerightrouting

11:

ℓ←∑cqϕ\(c∣𝐱\)ℓc\+γDKL\(qϕ\(c∣𝐱\)∥p\(c\)\)\\ell\\leftarrow\\sum\_\{c\}q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\\,\\ell\_\{c\}\+\\gamma\\,D\_\{\\mathrm\{KL\}\}\(q\_\{\\phi\}\(c\\mid\\mathbf\{x\}\)\\,\\\|\\,p\(c\)\)⊳\\trianglerightweighted loss,[Eq\. 17](https://arxiv.org/html/2610.33577#A2.E17)

12:update

ϕ,θ\\phi,\\thetaby gradient descent on

ℓ\\ell
13:endwhile

Algorithm 2Sampling from Variational Atlas Autoencoders\. Only the selected decoder is evaluated, so the cost is independent ofCC\. With uniformp⁡\(c\)p\(c\)and standard normal priors,p^​\(c\)=1/C\\hat\{p\}\(c\)=1/Cand\(𝐦c,𝐒c\)=\(𝟎,𝐈\)\(\\mathbf\{m\}\_\{c\},\\mathbf\{S\}\_\{c\}\)=\(\\mathbf\{0\},\\mathbf\{I\}\); the moment\-matched values provide an optional correction described in[Sec\. 3\.2](https://arxiv.org/html/2610.33577#S3.SS2)\.1:trained decoders

\{gc\}c=1C\\\{g\_\{c\}\\\}\_\{c=1\}^\{C\}, chart weights

p^​\(c\)\\hat\{p\}\(c\), sampling distributions

\{p~\(𝐳c∣c\)=𝒩\(𝐦c,𝐒c\)\}c=1C\\\{\\tilde\{p\}\(\\mathbf\{z\}\_\{c\}\\mid c\)=\\mathcal\{N\}\(\\mathbf\{m\}\_\{c\},\\mathbf\{S\}\_\{c\}\)\\\}\_\{c=1\}^\{C\}
2:

c∼p^​\(c\)c\\sim\\hat\{p\}\(c\)⊳\\trianglerightpick a chart

3:

𝐳c∼p~​\(𝐳c∣c\)\\mathbf\{z\}\_\{c\}\\sim\\tilde\{p\}\(\\mathbf\{z\}\_\{c\}\\mid c\)⊳\\trianglerightsample its latent

4:

𝐱^←gc​\(𝐳c\)\\hat\{\\mathbf\{x\}\}\\leftarrow g\_\{c\}\(\\mathbf\{z\}\_\{c\}\)⊳\\trianglerightdecode

5:return

𝐱^\\hat\{\\mathbf\{x\}\}

Algorithm[1](https://arxiv.org/html/2610.33577#alg1)and[2](https://arxiv.org/html/2610.33577#alg2)summarize the training and sampling procedures\. During training, every encoder is evaluated on each minibatch\. The encoder outputs determine both the routing weights via the latent\-evidence score of[Eq\. 7](https://arxiv.org/html/2610.33577#S3.E7)and, through a reparameterized sample, the per\-chart losses of[Eq\. 16](https://arxiv.org/html/2610.33577#A2.E16)\. These are combined into the weighted objective of[Eq\. 17](https://arxiv.org/html/2610.33577#A2.E17), and all encoders and decoders are updated jointly\. Sampling requires only a chart index, a latent sample from that chart’s prior, and a single decoder pass\.

## Appendix FAdditional results

We present additional results for 10\-chart variants of the variational atlas autoencoder trained on MNIST, Fashion\-MNIST, and CelebA\.[Figure 7](https://arxiv.org/html/2610.33577#A6.F7)depicts training samples and their reconstructions from the fully converged model\.[Figure 9](https://arxiv.org/html/2610.33577#A6.F9)depicts samples drawn from each of the 10 charts within each VAA\.

Table 5:Sample FID \(mean±\\pmstd over seeds\) by routing function, DCGAN backbone,C=10C=10charts, and iteration counts matched within each dataset\. Lower is better; best in bold, second best underlined\.Latent\-evidence routingDatasetELBOμ\\mu\-basedResamplingExpected log\-priorPrior\-marginalMNIST11\.80±\\pm0\.1012\.99±\\pm0\.2213\.79±\\pm1\.0012\.77±\\pm0\.3312\.62±\\pm0\.53Fashion\-MNIST43\.64±\\pm0\.3438\.42±\\pm0\.4939\.77±\\pm1\.4638\.03±\\pm0\.6438\.53±\\pm0\.45CelebA47\.14±\\pm0\.1943\.46±\\pm0\.5344\.05±\\pm0\.2443\.89±\\pm0\.5543\.69±\\pm0\.24#### Choice of routing probability\.

[Table 5](https://arxiv.org/html/2610.33577#A6.T5)shows the effect of the routing function on the sample FID averaged over three seeds\. The different variants of latent\-evidence routing produce the best outcome in most cases\. For our experiments, we choose prior\-marginal routing, but we find that bothμ\\mu\-based and expected log\-prior routing produce comparable results\.

#### Reconstruction error\.

[Figure 7](https://arxiv.org/html/2610.33577#A6.F7)depicts additional reconstruction samples computed with our 10\-chart VAA and compares them with those from the capacity\-matched VAE\.

![Refer to caption](https://arxiv.org/html/2610.33577v1/mnist_control_vs_10_recon.png)\(a\)MNIST
![Refer to caption](https://arxiv.org/html/2610.33577v1/fashion_control_vs_10_recon.png)\(b\)Fashion\-MNIST
![Refer to caption](https://arxiv.org/html/2610.33577v1/celeba_control_vs_10_recon.png)\(c\)CelebA

Figure 7:10\-chart Variational Atlas Autoencoders \(ours\) and 1\-chart capacity\-matched VAE reconstructions of randomly selected held\-out inputs from MNIST, Fashion\-MNIST and CelebA\.![Refer to caption](https://arxiv.org/html/2610.33577v1/fig_torus_portal_interp.png)Figure 8:Interpolation trajectories between two charts through a portal on the double\-torus geometry\. For any two points on the torus, we can create interpolation trajectories in our charts’ latent spaces that remain close to the torus surface \(the data manifold\)\. The red line illustrates straight\-line interpolation through a global latent space, which leaves the surface and thus passes through implausible points\. Optimizing portal placement to select shorter paths is left as future work\.
#### Generated samples\.

[Figure 9](https://arxiv.org/html/2610.33577#A6.F9)includes additional samples generated from each of our charts and from a capacity\-matched vanilla VAE baseline\. Samples are drawn from each chart’s prior at random\. Sampling from our VAA model, only requires the sampled chart’s weights to be active, resulting in faster sampling compared to our baselines which need all the model parameters\.

![Refer to caption](https://arxiv.org/html/2610.33577v1/mnist_chart_samples_vs_control_5x5.png)\(a\)MNIST: 10 charts
![Refer to caption](https://arxiv.org/html/2610.33577v1/celeba_lpips_chart_samples_vs_control_5x5.png)\(b\)CelebA: 10 charts
![Refer to caption](https://arxiv.org/html/2610.33577v1/fashion_20_charts_samples_vs_vae_5x5.png)\(c\)Fashion\-MNIST: 20 charts

Figure 9:Sample images drawn from 10\-, 10\-, and 20\-chart VAAs trained on MNIST, CelebA, and Fashion\-MNIST, respectively\. Each grid represents samples generated from a distinct chart\. Samples from a capacity\-matched VAE are shown on the right\.
#### Geometric interpolation results\.

[Figure 8](https://arxiv.org/html/2610.33577#A6.F8)shows interpolation trajectories over the double torus, including paths across chart boundaries through portals\.