原生几何与线性可恢复性：对齐如何改变柏拉图式扩展

# Native Geometry vs. Linear Recoverability: How Alignment Changes Platonic Scaling
Source: [https://arxiv.org/html/2610.22749](https://arxiv.org/html/2610.22749)
\\workshoptitle

Symmetry and Geometry in Neural Representations \(NeurReps\)

Austin LutterbachAffiliation:EleutherAI SOARAffiliation:UniverseTBDEmail:[austin\.lutterbach@proton\.me](mailto:)Jasper ChongAffiliation:University of QueenslandAffiliation:EleutherAI SOAREmail:[jasper\.chong@uq\.edu\.au](mailto:)Aman KumarAffiliation:UniverseTBDAffiliation:EleutherAIAffiliation:IUCAAEmail:[kumaramanasci@gmail\.com](mailto:)Michael J\. SmithAffiliation:AstroAIAffiliation:Harvard\-Smithsonian Center for AstrophysicsAffiliation:UniverseTBDAffiliation:EleutherAIEmail:[mike@mjjsmith\.com](mailto:)

###### Abstract

Do larger models learn increasingly similar representations of the same world? The Platonic Representation Hypothesis \(PRH\) predicts that they should, but “similar” can mean native cosine geometry—nearest\-neighbour structure induced by cosine similarity before any learned alignment—or linear recoverability: correspondence revealed by a supervised linear map\. Using matched Legacy Survey and Hyper Suprime\-Cam \(HSC\) embeddings across five model families, we find that Ridge alignment strengthens the relationship between model size and cross\-dataset neighbourhood correspondence relative to native cosine geometry\. The effect appears in all five families and persists when both representations are reduced to dimension256256, so it is not simply that wider embeddings give alignment more dimensions to work with\. The learned maps remain far from rotations or uniform scaling, and Sparse Autoencoder \(SAE\) and Block\-Sparse Featuriser \(BSF\) features give no advantage over the same alignment applied to dense embeddings\. Model scale thus improves linear recoverability more clearly than native cosine\-neighbourhood correspondence, without establishing convergence toward a shared native metric geometry\.

## 1Introduction

Representations from independently trained models often become more similar with increasing capability and scale\([Huh et al\., 2024](https://arxiv.org/html/2610.22749#bib.bib1)\)\. The Platonic Representation Hypothesis \(PRH\) interprets this as convergence toward a shared statistical model of reality\. Astronomical surveys provide a useful testbed: the Legacy Survey\([Dey et al\., 2019](https://arxiv.org/html/2610.22749#bib.bib12)\)and Hyper Suprime\-Cam \(HSC\)\([Miyazaki et al\., 2018](https://arxiv.org/html/2610.22749#bib.bib11)\)observe the same physical objects through distinct depths, resolutions, and filter sets, and prior work reports increasing mutualkk\-nearest\-neighbour \(mKNN\) agreement between their embeddings as capacity grows\([UniverseTBD et al\., 2025](https://arxiv.org/html/2610.22749#bib.bib2)\)\.

Two representations may become intrinsically more geometrically similar as models scale, or they may remain geometrically different while becoming easier to translate between with a learned map\. We distinguish these possibilities\. Prior work studies alignment through supervised linear maps\([Mikolov et al\., 2013](https://arxiv.org/html/2610.22749#bib.bib13)\)and unpaired embedding translation\([Lample et al\., 2018](https://arxiv.org/html/2610.22749#bib.bib14);[Moschella et al\., 2023](https://arxiv.org/html/2610.22749#bib.bib15);[Jha et al\., 2025](https://arxiv.org/html/2610.22749#bib.bib4)\)\. Here*native cosine geometry*is the nearest\-neighbour structure induced directly by cosine similarity before any learned alignment, and*linear recoverability*is the correspondence exposed after fitting a supervised affine/Ridge map on paired training objects \(*linear alignment*\)\. An affine map can improve neighbourhood correspondence without preserving distances or angles, so stronger recoverability need not imply a shared native metric geometry\. Terminology is summarized in Appendix[A](https://arxiv.org/html/2610.22749#A1)\.

On the same held\-out objects we measure meankk\-nearest\-neighbour agreement before and after linear alignment,

Mnative≡MDense,Mlinear≡MDense\+Ridge,M\_\{\\mathrm\{native\}\}\\equiv M\_\{\\mathrm\{Dense\}\},\\qquad M\_\{\\mathrm\{linear\}\}\\equiv M\_\{\\mathrm\{Dense\+Ridge\}\},with alignment gainG=Mlinear−MnativeG=M\_\{\\mathrm\{linear\}\}\-M\_\{\\mathrm\{native\}\}\(Section[2](https://arxiv.org/html/2610.22749#S2)\)\.

We report three main findings\.\(1\)Linear recoverability scales more strongly with model size than native correspondence\.\(2\)The effect persists after independently projecting both surveys to256256principal components, so increased alignment dimensionality alone does not explain it\.\(3\)The maps exposing this additional correspondence remain strongly anisotropic—scaling different directions by different amounts—while most of the mKNN gain is carried by a much smaller singular subspace\. Secondary PCA, SAE/BSF\([Gao et al\., 2025](https://arxiv.org/html/2610.22749#bib.bib5);[Fel et al\., 2026](https://arxiv.org/html/2610.22749#bib.bib3)\), same\-dataset cross\-model, and unpaired nonlinear controls are reported in the appendix\.111[https://github\.com/hypervalis/native\-geometry\-linear\-recoverability](https://github.com/hypervalis/native-geometry-linear-recoverability)

## 2Setup

We use the official UniverseTBD Legacy↔\\leftrightarrowHSC matched embeddings\([UniverseTBD et al\., 2025](https://arxiv.org/html/2610.22749#bib.bib2)\)\(n=16,384n\{=\}16\{,\}384objects\) across five families—AstroPTv2\([Smith et al\., 2024](https://arxiv.org/html/2610.22749#bib.bib10)\), ConvNeXtv2\([Woo et al\., 2023](https://arxiv.org/html/2610.22749#bib.bib7)\), DINOv2\([Oquab et al\., 2024](https://arxiv.org/html/2610.22749#bib.bib8)\), ViT\([Dosovitskiy et al\., 2021](https://arxiv.org/html/2610.22749#bib.bib6)\), and I\-JEPA\([Assran et al\., 2023](https://arxiv.org/html/2610.22749#bib.bib9)\)—sixteen size tiers in total \(Appendix[B](https://arxiv.org/html/2610.22749#A2)\), with only two tiers for I\-JEPA\.

LetNkA​\(i\)N\_\{k\}^\{A\}\(i\)andNkB​\(i\)N\_\{k\}^\{B\}\(i\)be thekk\-nearest\-neighbour sets of objectiiunder cosine similarity in representationsAAandBB\(excludingii\)\. Our observable is the mutualkk\-nearest\-neighbour score

Mk=1n​∑i=1n\|NkA​\(i\)∩NkB​\(i\)\|k,M\_\{k\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\frac\{\\bigl\|N\_\{k\}^\{A\}\(i\)\\cap N\_\{k\}^\{B\}\(i\)\\bigr\|\}\{k\},soMkM\_\{k\}is mean neighbourhood overlap rather than a direct cross\-space cosine comparison\([Huh et al\., 2024](https://arxiv.org/html/2610.22749#bib.bib1);[UniverseTBD et al\., 2025](https://arxiv.org/html/2610.22749#bib.bib2)\);Mk=1M\_\{k\}\{=\}1for identical neighbourhoods\. We usek=10k\{=\}10as the primary setting and write this as mKNN@10\.

All primary results use a frozen train/test split \(ntrain=13,107n\_\{\\mathrm\{train\}\}\{=\}13\{,\}107,ntest=3277n\_\{\\mathrm\{test\}\}\{=\}3277, seed00\) and atest\-only gallery:MkM\_\{k\}is averaged over held\-out objects, and neighbours are sought only among held\-out objects, so chance isM10≈k/\(ntest−1\)≈0\.0031M\_\{10\}\\approx k/\(n\_\{\\mathrm\{test\}\}\{\-\}1\)\\approx 0\.0031\.

We index model size byPP, the approximate parameter count in millions \(Appendix[B](https://arxiv.org/html/2610.22749#A2)\), and writeXXandYYfor the paired Legacy and HSC embedding matrices\. Direction is Legacy→\\toHSC\. We fit an affine Ridge predictor on paired training objects after train\-only standardization \(StandardScaler\\mathrm\{StandardScaler\}onX,YX,Y, then Ridge withα=1\\alpha\{=\}1and an intercept\), mapping Legacy into HSC space\. Geometric diagnostics characterise its linear componentAeffA\_\{\\mathrm\{eff\}\}in original coordinates \(Appendix[E](https://arxiv.org/html/2610.22749#A5)\)\.

## 3Linear recoverability scales more strongly than native correspondence

The*level*of cross\-survey correspondence and how that correspondence changes with model size are distinct\. We also test whether sparse feature spaces expose more shared representational structure by applying the same Ridge alignment to SAE and BSF features\. We compare native dense embeddings \(Dense\), Ridge\-aligned dense embeddings \(Dense\+Ridge\), and the same Ridge procedure applied to SAE and BSF features \(SAE\+Ridge and BSF\+Ridge\)\. Relative to native Dense, SAE\+Ridge and BSF\+Ridge raise held\-out mKNN@10 from0\.0220\.022to0\.0330\.033and0\.0350\.035, but neither consistently steepens the size slope, and both remain below Dense\+Ridge \(0\.0420\.042; Appendix[D](https://arxiv.org/html/2610.22749#A4)\)\. The interesting Dense\+Ridge result is therefore not merely that Ridge improves mKNN, but that the improvement grows with model size\.

For Legacy→\\toHSC, Dense\+Ridge raises mean held\-out mKNN@10 from0\.0220\.022to0\.0420\.042\(G≈0\.020G\\approx 0\.020; Figure[1](https://arxiv.org/html/2610.22749#S3.F1)a\)—roughly twice native overlap and about13×13\\timeschance \(≈0\.0031\\approx 0\.0031\)\. For each architecture familyFF, fitting mKNN againstlog10⁡P\\log\_\{10\}Pgives

Δ​βF=βDense\+Ridge,F−βDense,F\.\\Delta\\beta\_\{F\}=\\beta\_\{\\mathrm\{Dense\+Ridge\},F\}\-\\beta\_\{\\mathrm\{Dense\},F\}\.We findΔ​βF\>0\\Delta\\beta\_\{F\}\>0in all five families, and the equal\-family mean

T=15​∑FΔ​βF=0\.00721T=\\frac\{1\}\{5\}\\sum\_\{F\}\\Delta\\beta\_\{F\}=0\.00721shows that correspondence exposed by alignment grows faster with size than native correspondence does\. Averaging families equally makes architecture family, not checkpoint count, the replication unit\. The effect is positive in all five families \(psign=0\.031p\_\{\\mathrm\{sign\}\}=0\.031\), remains positive under held\-out\-object bootstrap \(95% CI\[0\.0034,0\.0111\]\[0\.0034,0\.0111\]\), and exceeds a synchronized correspondence\-shuffle refit null \(pperm≈0\.005p\_\{\\mathrm\{perm\}\}\\approx 0\.005; Appendix[B](https://arxiv.org/html/2610.22749#A2)\)\. It is stable acrossk∈\{5,10,25,50\}k\\in\\\{5,10,25,50\\\}\. HSC→\\toLegacy is weaker and more heterogeneous, so we treat linear recoverability as directional rather than as a symmetric equivalence \(Appendix[C](https://arxiv.org/html/2610.22749#A3)\)\.

![Refer to caption](https://arxiv.org/html/2610.22749v1/figures/scaling_dense_and_fixedrank.png)Figure 1:Held\-out mKNN@10 versuslog10⁡P\\log\_\{10\}P\(PP: parameter count in millions; Appendix[B](https://arxiv.org/html/2610.22749#A2)\)\. Curves compare native Dense correspondence with Dense\+Ridge linear recoverability\.\(a\)Original embedding dimensions: Ridge raises both the level and the size slope \(T=0\.00721T=0\.00721\)\.\(b\)Independent PCA to256256dimensions per survey \(same feature dimension in both conditions\): the slope amplification persists \(T256=0\.00864T\_\{256\}=0\.00864\)\.
## 4Recoverability is concentrated within broadly anisotropic maps

Scaled\-orthogonal mapsx↦c​Q​xx\\mapsto cQxpreserve pairwise cosine similarities, so they cannot change cosine mKNN\. The observed gain must therefore come from non\-geometry\-preserving parts of the fitted affine map, potentially including translation\.

LetWWbe the fitted Ridge coefficient matrix andDx=diag⁡\(σx\)D\_\{x\}=\\mathrm\{diag\}\(\\sigma\_\{x\}\),Dy=diag⁡\(σy\)D\_\{y\}=\\mathrm\{diag\}\(\\sigma\_\{y\}\)the train\-feature standard deviations\. The effective linear component in original coordinates is

Aeff=Dy​W​Dx−1=U​Σ​V⊤,Σ=diag⁡\(s1,…,sd\),A\_\{\\mathrm\{eff\}\}=D\_\{y\}WD\_\{x\}^\{\-1\}=U\\Sigma V^\{\\top\},\\qquad\\Sigma=\\mathrm\{diag\}\(s\_\{1\},\\ldots,s\_\{d\}\),with singular valuessi\>0s\_\{i\}\>0\. Direction\-dependent stretch is summarized by

Alog=stdi​\(log⁡si\),A\_\{\\log\}=\\mathrm\{std\}\_\{i\}\(\\log s\_\{i\}\),which ranges from1\.761\.76–3\.153\.15across tiers: alignment requires substantial anisotropic stretch rather than rotation/reflection and global rescaling alone\.

How broadly is that anisotropy distributed? With centered log\-stretchesai=log⁡si−1d​∑jlog⁡sja\_\{i\}=\\log s\_\{i\}\-\\tfrac\{1\}\{d\}\\sum\_\{j\}\\log s\_\{j\}and anisotropy massesmi=ai2m\_\{i\}=a\_\{i\}^\{2\}, letf90anisf\_\{90\}^\{\\mathrm\{anis\}\}be the smallest fraction of singular directions \(ordered by decreasingmim\_\{i\}\) needed for90%90\\%of∑imi\\sum\_\{i\}m\_\{i\}\. Across the1616maps,f90anisf\_\{90\}^\{\\mathrm\{anis\}\}is0\.4110\.411–0\.4900\.490\(median0\.4270\.427\): departure from uniform scaling is spread across a substantial fraction of the representation\.

![Refer to caption](https://arxiv.org/html/2610.22749v1/figures/geometry_anisotropy_vs_functional.png)Figure 2:Filled markers: fraction ofAeffA\_\{\\mathrm\{eff\}\}singular directions needed to recover90%90\\%of the Dense\+Ridge mKNN lift \(median4\.8%4\.8\\%\)\. Open markers: fraction needed for90%90\\%of the map’s anisotropy mass \(median42\.7%42\.7\\%\)\. The map is anisotropic across many directions, but most neighbour recovery is concentrated in a much smaller subspace\.TruncatingAeffA\_\{\\mathrm\{eff\}\}along singular directions without refitting recovers90%90\\%of the Dense\+Ridge mKNN liftG=MDense\+Ridge−MnativeG=M\_\{\\mathrm\{Dense\+Ridge\}\}\-M\_\{\\mathrm\{native\}\}with only2\.12\.1–17%17\\%of directions \(median4\.8%4\.8\\%; Figure[2](https://arxiv.org/html/2610.22749#S4.F2)\)\. Spectral anisotropy and functional recoverability therefore have sharply different effective dimensionalities\.

This contrast does not itself show a compelling size trend:f90anisf\_\{90\}^\{\\mathrm\{anis\}\}decreases slightly in four of five families, but only by a few percentage points, and the pooled association withlog10⁡P\\log\_\{10\}Pis weak \(descriptive Spearmanρ=0\.13\\rho\{=\}0\.13,p=0\.64p\{=\}0\.64,n=16n\{=\}16\);AlogA\_\{\\log\}andf90anisf\_\{90\}^\{\\mathrm\{anis\}\}are likewise weakly associated \(ρ=0\.39\\rho\{=\}0\.39,p=0\.13p\{=\}0\.13; Appendix[E](https://arxiv.org/html/2610.22749#A5)\)\. The part ofAeffA\_\{\\mathrm\{eff\}\}that matters for cosine\-neighbour recovery remains functionally low\-dimensional\.

## 5Discussion and conclusion

Apparent representational convergence depends on the transformation class used to compare representations\. Native cosine correspondence increases only weakly with model size, and scaled\-orthogonal maps leave cosine neighbourhoods unchanged\. Under supervised affine/Ridge alignment the size–correspondence relationship is stronger \(T=0\.00721T\{=\}0\.00721;5/55/5families\) and survives fixed independent\-PCA dimension256256\(T256=0\.00864T\_\{256\}\{=\}0\.00864;5/55/5\)\. The learned maps remain anisotropic, and translation ablation leaves most of the gain and slope amplification intact, so the effect is carried primarily byAeffA\_\{\\mathrm\{eff\}\}rather than by an intercept \(Appendix[E](https://arxiv.org/html/2610.22749#A5)\)\. Reverse amplification is weaker \(TH→L=0\.00447T\_\{H\\to L\}\{=\}0\.00447\), and a same\-dataset cross\-model control retainsGA→B≠GB→AG\_\{A\\to B\}\\neq G\_\{B\\to A\}\(Appendix[C](https://arxiv.org/html/2610.22749#A3),[F](https://arxiv.org/html/2610.22749#A6)\)\. Limitations include paired Ridge supervision, five families, one astronomy setting, and a frozen split;TTis stable across Ridgeα\\alphaandkk\(Appendix[B](https://arxiv.org/html/2610.22749#A2),[C](https://arxiv.org/html/2610.22749#A3)\)\. Scaling strengthens supervised linear recoverability more clearly than native cosine\-neighbourhood correspondence, without establishing shared\-metric convergence\.

The character of this correspondence is not strictly Euclidean geometric convergence: two representation spaces may be different perspectives on the same objects, related by a learned transformation\. While affine Ridge mappings improve mKNN–size scaling substantially, absolute mKNN remains low\. One possibility is that the scaling relationship becomes substantially stronger at larger sizes\. Another is that the true recovery map is only weakly approximated by a global affine map—more complicated or ill\-conditioned\. If the underlying relationship is sufficiently smooth, a natural hypothesis is that within small regions of representation space it can be approximated by a local affine map, potentially revealing sharper Platonic scaling\. Testing that would help determine whether limited global alignment reflects genuinely different native geometries or the inadequacy of a single affine transform\.

## Acknowledgements

We would like to thank Stella Biderman and EleutherAI for their compute and infrastructure support\.

## References

- Assranet al\.\(2023\)M\. Assran, Q\. Duval, I\. Misra, P\. Bojanowski, P\. Vincent, M\. Rabbat, Y\. LeCun, and N\. BallasSelf\-supervised learning from images with a joint\-embedding predictive architecture\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 15619–15629\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52729.2023.01499)Cited by:[§2](https://arxiv.org/html/2610.22749#S2.p1.1)\.
- Deyet al\.\(2019\)A\. Dey, D\. J\. Schlegel, D\. Lang, R\. Blum, K\. Burleigh, X\. Fan, J\. R\. Findlay, D\. Finkbeiner, D\. Herrera, S\. Juneau,et al\.Overview of the DESI legacy imaging surveys\.The Astronomical Journal157\(5\),pp\. 168\.External Links:[Document](https://dx.doi.org/10.3847/1538-3881/ab089d)Cited by:[§1](https://arxiv.org/html/2610.22749#S1.p1.1)\.
- Dosovitskiyet al\.\(2021\)A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly, J\. Uszkoreit, and N\. HoulsbyAn image is worth 16x16 words: transformers for image recognition at scale\.InInternational Conference on Learning Representations,External Links:2010\.11929Cited by:[§2](https://arxiv.org/html/2610.22749#S2.p1.1)\.
- Felet al\.\(2026\)T\. Fel, M\. Kowal, M\. Jacobs, D\. Hazra, U\. Bhalla, L\. Sharkey, L\. Bushnaq, S\. Grant, T\. Haklay, T\. Icard, C\. Rager, M\. Pearce, D\. Wurgaft, A\. Swann, F\. Doshi, S\. Boppana, C\. Tigges, N\. Cammarata, T\. Serre, V\. Shyam, O\. Lewis, T\. McGrath, J\. Merullo, E\. S\. Lubana, and A\. GeigerStructuring sparsity: block\-sparse featurizers capture visual concept manifolds\.arXiv preprint arXiv:2606\.25234\.External Links:2606\.25234Cited by:[Appendix D](https://arxiv.org/html/2610.22749#A4.p2.1),[§1](https://arxiv.org/html/2610.22749#S1.p4.1)\.
- Gaoet al\.\(2025\)L\. Gao, T\. Dupré la Tour, H\. Tillman, G\. Goh, R\. Troll, A\. Radford, I\. Sutskever, J\. Leike, and J\. WuScaling and evaluating sparse autoencoders\.InInternational Conference on Learning Representations,External Links:2406\.04093,[Link](https://arxiv.org/abs/2406.04093)Cited by:[Appendix D](https://arxiv.org/html/2610.22749#A4.p2.1),[§1](https://arxiv.org/html/2610.22749#S1.p4.1)\.
- Huhet al\.\(2024\)M\. Huh, B\. Cheung, T\. Wang, and P\. IsolaPosition: the Platonic representation hypothesis\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 20617–20642\.External Links:[Link](https://proceedings.mlr.press/v235/huh24a.html),2405\.07987Cited by:[§1](https://arxiv.org/html/2610.22749#S1.p1.1),[§2](https://arxiv.org/html/2610.22749#S2.p2.2)\.
- Jhaet al\.\(2025\)R\. Jha, C\. Zhang, V\. Shmatikov, and J\. X\. MorrisHarnessing the universal geometry of embeddings\.InAdvances in Neural Information Processing Systems,External Links:2505\.12540,[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/4175dee33d6145cb8f0323703d138a53-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2610.22749#S1.p2.1)\.
- Lampleet al\.\(2018\)G\. Lample, A\. Conneau, M\. Ranzato, L\. Denoyer, and H\. JégouWord translation without parallel data\.InInternational Conference on Learning Representations,External Links:1710\.04087,[Link](https://openreview.net/forum?id=H196sainb)Cited by:[§1](https://arxiv.org/html/2610.22749#S1.p2.1)\.
- Mikolovet al\.\(2013\)T\. Mikolov, Q\. V\. Le, and I\. SutskeverExploiting similarities among languages for machine translation\.arXiv preprint arXiv:1309\.4168\.External Links:1309\.4168Cited by:[§1](https://arxiv.org/html/2610.22749#S1.p2.1)\.
- Miyazakiet al\.\(2018\)S\. Miyazaki, Y\. Komiyama, S\. Kawanomoto, Y\. Fujimori, T\. Hamana, H\. Hayashi, H\. Heiguchi, T\. Hattori, S\. Huang, M\. Imanishi,et al\.Hyper Suprime\-Cam: system design and verification of image quality\.Publications of the Astronomical Society of Japan70\(SP1\),pp\. S1\.External Links:[Document](https://dx.doi.org/10.1093/pasj/psx063)Cited by:[§1](https://arxiv.org/html/2610.22749#S1.p1.1)\.
- Moschellaet al\.\(2023\)L\. Moschella, V\. Maiorca, M\. Fumero, A\. Norelli, F\. Locatello, and E\. RodolàRelative representations enable zero\-shot latent space communication\.InInternational Conference on Learning Representations,External Links:2209\.15430,[Link](https://openreview.net/forum?id=SrC-nwieGJ)Cited by:[§1](https://arxiv.org/html/2610.22749#S1.p2.1)\.
- Oquabet al\.\(2024\)M\. Oquab, T\. Darcet, T\. Moutakanni, H\. V\. Vo, M\. Szafraniec, V\. Khalidov, P\. Fernandez, D\. Haziza, F\. Massa, A\. El\-Nouby, M\. Assran, N\. Ballas, W\. Galuba, R\. Howes, P\. Huang, S\. Li, I\. Misra, M\. Rabbat, V\. Sharma, G\. Synnaeve, H\. Xu, H\. Jégou, J\. Mairal, P\. Labatut, A\. Joulin, and P\. BojanowskiDINOv2: learning robust visual features without supervision\.Transactions on Machine Learning Research\.External Links:2304\.07193Cited by:[§2](https://arxiv.org/html/2610.22749#S2.p1.1)\.
- Smithet al\.\(2024\)M\. J\. Smith, R\. J\. Roberts, E\. Angeloudi, and M\. Huertas\-CompanyAstroPT: scaling large observation models for astronomy\.InICML 2024 AI for Science Workshop,External Links:2405\.14930Cited by:[§2](https://arxiv.org/html/2610.22749#S2.p1.1)\.
- UniverseTBDet al\.\(2025\)UniverseTBD, K\. Duraphe, M\. J\. Smith, S\. Sourav, and J\. F\. WuThe Platonic universe: do foundation models see the same sky?\.arXiv preprint arXiv:2509\.19453\.Note:Accepted to the NeurIPS 2025 Workshop on Machine Learning and the Physical SciencesExternal Links:2509\.19453,[Document](https://dx.doi.org/10.48550/arXiv.2509.19453)Cited by:[§1](https://arxiv.org/html/2610.22749#S1.p1.1),[§2](https://arxiv.org/html/2610.22749#S2.p1.1),[§2](https://arxiv.org/html/2610.22749#S2.p2.2)\.
- Wooet al\.\(2023\)S\. Woo, S\. Debnath, R\. Hu, X\. Chen, Z\. Liu, I\. S\. Kweon, and S\. XieConvNeXt V2: co\-designing and scaling ConvNets with masked autoencoders\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 16133–16142\.Cited by:[§2](https://arxiv.org/html/2610.22749#S2.p1.1)\.

## Appendix ATerminology and notation

### Native cosine geometry\.

Nearest\-neighbour structure induced directly by cosine similarity in the original embeddings, before any learned alignment\.

### Native correspondence\.

Cross\-representation agreement measured under native cosine geometry; in the headline analyses,Mnative≡MDenseM\_\{\\mathrm\{native\}\}\\equiv M\_\{\\mathrm\{Dense\}\}\.

### Linear alignment\.

A supervised affine/Ridge map fit on paired training objects that maps one representation into the other\.

### Linear recoverability\.

Correspondence exposed after linear alignment; in the headline analyses,Mlinear≡MDense\+RidgeM\_\{\\mathrm\{linear\}\}\\equiv M\_\{\\mathrm\{Dense\+Ridge\}\}\.

### Shared metric geometry / isometric convergence\.

The stronger claim that native geometries themselves converge toward the same metric structure \(distances/angles\), beyond correspondence under a learned map\.

### Dense / Dense\+Ridge / SAE\+Ridge / BSF\+Ridge\.

Representation–alignment conditions, not distinct similarity metrics:*Dense*= unaligned dense embeddings;*Dense\+Ridge*= Ridge on dense embeddings;*SAE\+Ridge*/*BSF\+Ridge*= the same Ridge recipe on SAE/BSF features\.

### mKNN / mKNN@10\.

Mutualkk\-nearest\-neighbour overlapMk=1n​∑i\|NkA​\(i\)∩NkB​\(i\)\|/kM\_\{k\}=\\tfrac\{1\}\{n\}\\sum\_\{i\}\|N\_\{k\}^\{A\}\(i\)\\cap N\_\{k\}^\{B\}\(i\)\|/k; primary settingk=10k\{=\}10\.

### AeffA\_\{\\mathrm\{eff\}\}\.

Effective linear component of the Ridge\-aligned map in original coordinates,Aeff=Dy​W​Dx−1A\_\{\\mathrm\{eff\}\}=D\_\{y\}WD\_\{x\}^\{\-1\}, whereWWis the standardized Ridge coefficient matrix andDx,DyD\_\{x\},D\_\{y\}hold train\-feature standard deviations\.

### Scaled\-orthogonal map\.

A mapx↦c​Q​xx\\mapsto cQxwithc\>0c\>0andQ⊤​Q=IQ^\{\\top\}Q=I; preserves pairwise cosine similarities and therefore cannot change cosine mKNN\.

### Anisotropy /AlogA\_\{\\log\}/f90anisf\_\{90\}^\{\\mathrm\{anis\}\}\.

Direction\-dependent stretching ofAeffA\_\{\\mathrm\{eff\}\}:Alog=stdi​\(log⁡si\)A\_\{\\log\}=\\mathrm\{std\}\_\{i\}\(\\log s\_\{i\}\)for singular valuessis\_\{i\};f90anisf\_\{90\}^\{\\mathrm\{anis\}\}is the fraction of singular directions needed for90%90\\%of the centered log\-stretch mass∑ai2\\sum a\_\{i\}^\{2\}\.

### Δ​βF\\Delta\\beta\_\{F\}/TT\.

For familyFF,Δ​βF=βDense\+Ridge,F−βDense,F\\Delta\\beta\_\{F\}=\\beta\_\{\\mathrm\{Dense\+Ridge\},F\}\-\\beta\_\{\\mathrm\{Dense\},F\}is the difference of OLS slopes of mKNN versuslog10⁡P\\log\_\{10\}P;T=15​∑FΔ​βFT=\\tfrac\{1\}\{5\}\\sum\_\{F\}\\Delta\\beta\_\{F\}is the equal\-family mean\.

## Appendix BProtocol and statistical robustness

Approximate parameter counts \(10610^\{6\}\): AstroPTv215/95/85015/95/850; ConvNeXtv2 nano/tiny/base/large15/28/89/19815/28/89/198; DINOv2 small/base/large/giant22/86/300/110022/86/300/1100; ViT base/large/huge86/307/63286/307/632; I\-JEPA huge/giant630/1000630/1000\.

The primary statisticT=15​∑FΔ​βFT=\\tfrac\{1\}\{5\}\\sum\_\{F\}\\Delta\\beta\_\{F\}averages family slope differences equally so that architecture family—not checkpoint count—is the replication unit\. The forward effect is positive in all five families \(psign=0\.03125p\_\{\\mathrm\{sign\}\}\{=\}0\.03125\), exceeds a synchronized correspondence\-shuffle refit null \(B=200B\{=\}200,p=1/201≈0\.005p\{=\}1/201\\approx 0\.005\), and remains positive under held\-out\-object bootstrap \(95% CI\[0\.0034,0\.0111\]\[0\.0034,0\.0111\]\); excluding I\-JEPA givesT−I=0\.00537T\_\{\-I\}\{=\}0\.00537with four/four families positive\. The result is also stable acrossα∈\{0\.01,0\.1,1,10,100\}\\alpha\\in\\\{0\.01,0\.1,1,10,100\\\}, withT⁡\(α\)∈\[0\.00721,0\.00848\]T\(\\alpha\)\\in\[0\.00721,0\.00848\]and5/55/5positive families throughout\.

## Appendix CDirection and neighbourhood\-size robustness

All checks reuse the frozen train/test split, Ridge recipe, and test\-only gallery\.

### Reverse direction\.

HSC→\\toLegacy yieldsTH→L=0\.00447T\_\{H\\to L\}\{=\}0\.00447with4/54/5families positive \(AstroPT negative\)\. Excluding I\-JEPA,TH→L\(−I\)=0\.00103T\_\{H\\to L\}^\{\(\-I\)\}\{=\}0\.00103\. Reverse amplification is present but weaker and family\-heterogeneous; it is not a second primary claim\.

### Neighbourhood size\.

Aggregate forward amplification is stable acrossk∈\{5,10,25,50\}k\\in\\\{5,10,25,50\\\}:T⁡\(k\)∈\{0\.00782,0\.00721,0\.00810,0\.00869\}T\(k\)\\in\\\{0\.00782,0\.00721,0\.00810,0\.00869\\\}, with5/55/5positive families atk=5,10k\{=\}5,10and4/54/5atk=25,50k\{=\}25,50\.

### Pooled interaction\.

On a tier\-by\-method table, a family\-fixed\-effects OLS specificationM=αF\+γF​log10​P\+δF​R\+θ⁡\(R​log10​P\)M=\\alpha\_\{F\}\+\\gamma\_\{F\}\\log\_\{10\}P\+\\delta\_\{F\}R\+\\theta\(R\\log\_\{10\}P\), whereR=1R\{=\}1for Ridge\-aligned andR=0R\{=\}0for native observations, gives the same positive interaction direction \(θ=0\.00451\\theta\{=\}0\.00451\), providing a descriptive robustness check under a different weighting\. We do not treat ordinary IID OLS uncertainties for this pooled model as primary inference, because Dense/Ridge observations are paired by tier\. Equal\-familyTT, the family sign test, the synchronized shuffle\-refit null, and the object bootstrap remain the primary evidence\.

## Appendix DSecondary representation controls

*Independent*PCA\+Ridge raises absolute recoverability at mid/high ranks \(means≈0\.044/0\.050/0\.055\\approx 0\.044/0\.050/0\.055at128/256/512128/256/512\); the fixed\-rank256256result is reported in the main text\. A pooled train\-only PCA basis without pairing leaves native mKNN near Dense \(≈0\.024\\approx 0\.024at rank256256\), showing that dimensional reduction alone does not substantially increase native correspondence\. PCA does not yield a consistent additional scale interaction beyond Dense\+Ridge\.

Under matched Ridge supervision, SAE and BSF underperform Dense\+Ridge by0\.00850\.0085and0\.00620\.0062mKNN on average, respectively; family\-bootstrap 95% CIs are\[−0\.0120,−0\.0043\]\[\-0\.0120,\-0\.0043\]and\[−0\.0104,−0\.0007\]\[\-0\.0104,\-0\.0007\]\. Sparse coding therefore provides no additional cross\-survey correspondence once alignment supervision is controlled\. SAE protocol: dictionary width20482048, seed00, TopK from available checkpoints, train\-only Ridge and inverse\-document\-frequency \(IDF\) weighting, fixed Legacy→\\toHSC\([Gao et al\., 2025](https://arxiv.org/html/2610.22749#bib.bib5)\); BSF uses Vanilla signed codes with the same Ridge recipe\([Fel et al\., 2026](https://arxiv.org/html/2610.22749#bib.bib3)\)\. Global Ridge residuals remain weakly locally structured in source space \(k∈\{10,25,50\}k\\in\\\{10,25,50\\\}; all16×316\\times 3residual\-permutation tests givep=1/201p\{=\}1/201\), but absolute residual cosine similarity is≈0\.01\\approx 0\.01and mean relative held\-out MSE reduction from a patch intercept is≪0\.1%\\ll 0\.1\\%; we do not fit local maps\.

## Appendix EGeometry details

After train\-onlyStandardScaler\\mathrm\{StandardScaler\}, Ridge yieldsWWand an intercept in standardized coordinates\. We analyze the linear componentAeff=diag⁡\(σy\)​W​diag​\(1/σx\)A\_\{\\mathrm\{eff\}\}=\\mathrm\{diag\}\(\\sigma\_\{y\}\)W\\mathrm\{diag\}\(1/\\sigma\_\{x\}\)in original coordinates, whereμx,σx\\mu\_\{x\},\\sigma\_\{x\}andμy,σy\\mu\_\{y\},\\sigma\_\{y\}are the train\-only per\-feature means and standard deviations; the corresponding affine map isy^=Aeff​x\+beff\\hat\{y\}=A\_\{\\mathrm\{eff\}\}x\+b\_\{\\mathrm\{eff\}\}withbeff=μy−Aeff​μx\+Dy​bstdb\_\{\\mathrm\{eff\}\}=\\mu\_\{y\}\-A\_\{\\mathrm\{eff\}\}\\mu\_\{x\}\+D\_\{y\}b\_\{\\mathrm\{std\}\},Dy=diag⁡\(σy\)D\_\{y\}=\\mathrm\{diag\}\(\\sigma\_\{y\}\)andbstdb\_\{\\mathrm\{std\}\}the Ridge intercept in standardized coordinates\. WritingAeff=U​Σ​V⊤A\_\{\\mathrm\{eff\}\}=U\\Sigma V^\{\\top\}, the distance to the nearest similarity transform isDsim=minc,Q⊤​Q=I⁡‖Aeff−c​Q‖F/‖Aeff‖FD\_\{\\mathrm\{sim\}\}=\\min\_\{c,Q^\{\\top\}Q=I\}\\\|A\_\{\\mathrm\{eff\}\}\-cQ\\\|\_\{F\}/\\\|A\_\{\\mathrm\{eff\}\}\\\|\_\{F\}\(main\-text valuesDsim≈0\.88D\_\{\\mathrm\{sim\}\}\\approx 0\.88–0\.910\.91\)\. Because cosine mKNN is not translation\-invariant, we ablatebeffb\_\{\\mathrm\{eff\}\}without refitting: for each tier we compare nativeM⁡\(X,Y\)M\(X,Y\), full affineM⁡\(Aeff​X\+beff,Y\)M\(A\_\{\\mathrm\{eff\}\}X\+b\_\{\\mathrm\{eff\}\},Y\), and linear\-onlyM⁡\(Aeff​X,Y\)M\(A\_\{\\mathrm\{eff\}\}X,Y\)on the same test queries/gallery\. Affine reconstruction matches the sklearn pipeline tomax⁡\|err\|≤3\.5×10−5\\max\|\{\\rm err\}\|\\leq 3\.5\{\\times\}10^\{\-5\}\. Across the1616tiers the fraction of affine gain retained by the linear component isηA=median​0\.990\\eta\_\{A\}=\\mathrm\{median\}\\,0\.990\(mean0\.9850\.985; range\[0\.920,1\.048\]\[0\.920,1\.048\]\)\. Family slope amplification without translation isTA=0\.00854T\_\{A\}\{=\}0\.00854\(5/55/5positive\) versusTaffine=0\.00721T\_\{\\mathrm\{affine\}\}\{=\}0\.00721, soTA/Taffine=1\.184T\_\{A\}/T\_\{\\mathrm\{affine\}\}\{=\}1\.184\(excluding I\-JEPA,TA\(−I\)=0\.00570T\_\{A\}^\{\(\-I\)\}\{=\}0\.00570\)\. A train\-mean\-centered diagnostic likewise preserves amplification \(Tcentered=0\.01360T\_\{\\mathrm\{centered\}\}\{=\}0\.01360; Figure[3](https://arxiv.org/html/2610.22749#A5.F3)\)\. Thus the headline scaling effect is primarily attributable to anisotropic linear deformation rather than affine translation, licensing the geometric analysis ofAeffA\_\{\\mathrm\{eff\}\}below\. On Legacy testkedgek\_\{\\mathrm\{edge\}\}\-NN edgesδi​j\\delta\_\{ij\}, local log\-stretch issi​j=log⁡\(‖Aeff​δi​j‖2/‖δi​j‖2\)s\_\{ij\}=\\log\(\\\|A\_\{\\mathrm\{eff\}\}\\delta\_\{ij\}\\\|\_\{2\}/\\\|\\delta\_\{ij\}\\\|\_\{2\}\), summarized byσlocal=std⁡\(si​j\)\\sigma\_\{\\mathrm\{local\}\}=\\mathrm\{std\}\(s\_\{ij\}\)\. Four of five family slopes ofσlocal\\sigma\_\{\\mathrm\{local\}\}versuslog10⁡P\\log\_\{10\}Pare negative, with DINOv2 approximately flat/slightly positive\. Two further diagnostics summarize the singular value spectrumσ1≥⋯≥σd\\sigma\_\{1\}\\geq\\cdots\\geq\\sigma\_\{d\}ofAeffA\_\{\\mathrm\{eff\}\}\. The log\-anisotropyAlog=std⁡\(log⁡σi\)A\_\{\\log\}=\\mathrm\{std\}\(\\log\\sigma\_\{i\}\)vanishes exactly when all singular values are equal, i\.e\. whenAeff=c​QA\_\{\\mathrm\{eff\}\}=cQfor somec\>0c\>0andQ⊤​Q=IQ^\{\\top\}Q=I; it therefore measures departure from𝒯scaled​orth\\mathcal\{T\}\_\{\\mathrm\{scaled\\,orth\}\}directly, without the optimal\-rotation fit required byDsimD\_\{\\mathrm\{sim\}\}\. The normalized spectral entropyHnorm=−∑ipilogpi/logdH\_\{\\mathrm\{norm\}\}=\-\\sum\_\{i\}p\_\{i\}\\log p\_\{i\}/\\log d, withpi=σi2/∑jσj2p\_\{i\}=\\sigma\_\{i\}^\{2\}/\\sum\_\{j\}\\sigma\_\{j\}^\{2\}, is11for a spectrally flat map and decreases as the map concentrates its action on fewer directions\. Both indicate strongly non\-flat effective maps at every tier\. Truncating the fitted effective map without refitting shows that roughly6%6\\%of singular directions recover at least90%90\\%of the Dense\+Ridge mKNN lift\.

![Refer to caption](https://arxiv.org/html/2610.22749v1/figures/translation_ablation_summary.png)Figure 3:Translation contributes little; the effect is carried by the linear componentAeffA\_\{\\mathrm\{eff\}\}\.\(a\)Per\-tier alignment gainGGover native mKNN, using the full affine map versus the linear component alone withbeffb\_\{\\mathrm\{eff\}\}removed\. Points near the dashedy=xy\{=\}xline indicate the translation contributes negligibly \(median retained fractionηA=0\.99\\eta\_\{A\}\{=\}0\.99\)\.\(b\)Family slope amplificationsΔ​βF\\Delta\\beta\_\{F\}under the two conditions; the model\-size amplification survives removal ofbeffb\_\{\\mathrm\{eff\}\}\(TA=0\.00854T\_\{A\}\{=\}0\.00854versusTaffine=0\.00721T\_\{\\mathrm\{affine\}\}\{=\}0\.00721\)\.
## Appendix FSame\-dataset cross\-model control

Cross\-survey directionality may reflect survey\-information differences, so same\-Legacy cross\-model alignment tests whether asymmetry persists when the observation channel is fixed\. We evaluate all101101valid unordered cross\-family pairs among1616Legacy models \(train\-only independent PCA256 per model; Ridgeα=1\\alpha\{=\}1; mKNN@10; test\-only gallery; frozen 80/20 split; both directions fit independently\)\. For unordered pair\{A,B\}\\\{A,B\\\},

Gsym=12​\(GA→B\+GB→A\),Adir=\|GA→B−GB→A\|,G\_\{\\mathrm\{sym\}\}=\\tfrac\{1\}\{2\}\(G\_\{A\\to B\}\+G\_\{B\\to A\}\),\\qquad A\_\{\\mathrm\{dir\}\}=\|G\_\{A\\to B\}\-G\_\{B\\to A\}\|,whereGA→B=MA→BRidge−Mnative​\(A,B\)G\_\{A\\to B\}=M^\{\\mathrm\{Ridge\}\}\_\{A\\to B\}\-M\_\{\\mathrm\{native\}\}\(A,B\)and likewise forGB→AG\_\{B\\to A\}\.

Mean native mKNN is0\.0500\.050\(chance≈0\.00305\\approx 0\.00305\)\. Ridge increases correspondence in both directions for101/101101/101pairs \(meanGsym=0\.060G\_\{\\mathrm\{sym\}\}\{=\}0\.060; meanAdir=0\.036A\_\{\\mathrm\{dir\}\}\{=\}0\.036; mean relative asymmetryArel≈0\.33A\_\{\\mathrm\{rel\}\}\{\\approx\}0\.33\)\. Mean directional gain is similar for larger→\\tosmaller \(≈0\.059\\approx 0\.059\) and smaller→\\tolarger \(≈0\.062\\approx 0\.062\) maps, so there is no systematic larger\-to\-smaller advantage\. Descriptively, writingβmin\\beta\_\{\\min\}for the OLS slope againstlog10\\log\_\{10\}of the*smaller*model’s parameter count, both mutual recoverability and directional asymmetry increase with that scale \(βmin​\(Gsym\)=\+0\.026\\beta\_\{\\min\}\(G\_\{\\mathrm\{sym\}\}\)\{=\}\{\+\}0\.026,βmin​\(Adir\)=\+0\.017\\beta\_\{\\min\}\(A\_\{\\mathrm\{dir\}\}\)\{=\}\{\+\}0\.017\); pair\-level trends are not treated inferentially because model endpoints recur across pairs\.

## Appendix GFixed\-dimensional control\.

A straightforward concern is that larger models often have wider embeddings, and therefore give Ridge a higher\-dimensional transformation space in which to fit the correspondence\. We test this directly by projecting each survey independently to rankr=256r=256using train\-only PCA\. The native and aligned conditions then use exactly the same 256\-dimensional features:MindPCA256M\_\{\\mathrm\{indPCA256\}\}compares them directly, whileMindPCA256\+RidgeM\_\{\\mathrm\{indPCA256\+Ridge\}\}applies the same supervised alignment\.

Defining

Δ​βF,256=βindPCA256\+Ridge,F−βindPCA256,F,T256=15​∑FΔ​βF,256,\\Delta\\beta\_\{F,256\}=\\beta\_\{\\mathrm\{indPCA256\+Ridge\},F\}\-\\beta\_\{\\mathrm\{indPCA256\},F\},\\qquad T\_\{256\}=\\frac\{1\}\{5\}\\sum\_\{F\}\\Delta\\beta\_\{F,256\},we obtainT256=0\.00864T\_\{256\}\{=\}0\.00864, again withΔ​βF,256\>0\\Delta\\beta\_\{F,256\}\>0in all five families \(T256\(−I\)=0\.00673T\_\{256\}^\{\(\-I\)\}\{=\}0\.00673excluding I\-JEPA; Figure[1](https://arxiv.org/html/2610.22749#S3.F1)b\)\. The scaling effect therefore survives when the alignment dimensionality is held fixed\. It is not simply a consequence of larger models giving Ridge more coordinates to work with\.

## Appendix HSAE TopK sensitivity

Trained checkpoints use heterogeneous TopK in1818–2323with no common training\-timeKKacross all1616tiers\. To test whether the sparse\-control result depends on sparsity level, we therefore repeat the frozen Legacy→\\toHSC protocol under*inference\-time*TopK overridesK∈\{10,20,40\}K\\in\\\{10,20,40\\\}on the preferred dictionary width20482048seed\-00checkpoints, holding dictionary width, split, Ridge fitting, mapping direction, IDF, and the test\-only gallery fixed\. Across this range the mean residualMSAE\+Ridge−MDense\+RidgeM\_\{\\mathrm\{SAE\+Ridge\}\}\-M\_\{\\mathrm\{Dense\+Ridge\}\}stays negative \(−0\.0159\-0\.0159,−0\.0085\-0\.0085,−0\.0077\-0\.0077\), family\-bootstrap 95% CIs exclude zero, and the equal\-family scaling interactionTSAE,KT\_\{\\mathrm\{SAE\},K\}remains non\-positive \(−0\.0133\-0\.0133,−0\.0099\-0\.0099,−0\.0079\-0\.0079; at most one family positive\)\. Thus the sparse result is not specific to the original TopK choice: across the tested inference sparsity range, SAE does not improve correspondence over the matched Dense\+Ridge baseline and does not yield a consistently stronger size trend \(Figure[4](https://arxiv.org/html/2610.22749#A8.F4)\)\.

![Refer to caption](https://arxiv.org/html/2610.22749v1/figures/sae_topk_sensitivity.png)Figure 4:Sparse coding does not overtake the dense baseline at any tested sparsity\.\(a\)Mean residualMSAE\+Ridge−MDense\+RidgeM\_\{\\mathrm\{SAE\+Ridge\}\}\-M\_\{\\mathrm\{Dense\+Ridge\}\}under inference\-time TopKK∈\{10,20,40\}K\\in\\\{10,20,40\\\}, with family\-bootstrap 95% CIs\. Negative values indicate SAE underperforms the matched dense baseline\.\(b\)Equal\-family scaling interactionTSAE,KT\_\{\\mathrm\{SAE\},K\}, with per\-family slope differencesΔ​βF,KSAE\\Delta\\beta^\{\\mathrm\{SAE\}\}\_\{F,K\}faint behind it\. Both level and size trend stay non\-positive acrossKK\.
## Appendix IUnpaired nonlinear control

An unpaired DualEncoder shared bottleneck \(model\-specific MLPs intoZ=256Z\{=\}256; reconstruction, RBF\-MMD, cycle, and Gram terms; no paired cross\-survey training objects\) yields held\-out relational mKNN@10 of0\.0120\.012–0\.0230\.023versus chance≈0\.0035\\approx 0\.0035, but size dependence is non\-monotonic on the available ConvNeXtv2 and DINOv2 size tiers\.