图像自蒸馏的数据密度缩放定律

# Data Density Scaling Laws for Image Self-Distillation
Source: [https://arxiv.org/html/2610.19013](https://arxiv.org/html/2610.19013)
Ani VanyanHakob TamazyanHrant KhachatrianAffiliation:YerevaNN and Yerevan State University, Yerevan, ArmeniaEmail:[\{alla,ani,hakob,hrant\}@yerevann\.com](mailto:)

###### Abstract

Current scaling law formulations suggest that increasing the number of unique samples in a dataset improves model performance, while repeated exposure to the same samples provides diminishing value\. In this work, we examine this assumption for image self\-distillation models\. Through systematic experiments across diverse datasets, namely Maxar, SATLAS, Waymo, nuPlan, Ego4D, ImageNet, and YFCC15M, we demonstrate that the effects of sample repetition on downstream performance are non\-trivial and dataset\-dependent\. We identify settings where repeatedly training on a smaller random subset can outperform training on a larger set of unique samples\. This finding highlights a limitation of existing scaling laws\. To address this gap, we propose a modified version of a Chinchilla\-style scaling law that explicitly accounts for two factors: the number of unique samples and repetition per sample\. This formulation provides a framework for understanding data–compute trade\-offs and improves prediction in some regimes where repetition plays a central role, although its benefits are not consistent across datasets\. Finally, we examine how subset selection and augmentation affect downstream performance\.

## 1Introduction

Recent progress in foundation models has been strongly associated with scaling both model size and dataset size\[[1](https://arxiv.org/html/2610.19013#bib.bib1),[2](https://arxiv.org/html/2610.19013#bib.bib2),[3](https://arxiv.org/html/2610.19013#bib.bib3)\]\. In language modeling, empirical scaling laws suggest that performance improves predictably as a function of model parameters, compute, and the number of unique training tokens\[[4](https://arxiv.org/html/2610.19013#bib.bib4),[3](https://arxiv.org/html/2610.19013#bib.bib3)\]\. Similar trends have motivated large\-scale representation learning in computer vision\[[5](https://arxiv.org/html/2610.19013#bib.bib5),[6](https://arxiv.org/html/2610.19013#bib.bib6),[7](https://arxiv.org/html/2610.19013#bib.bib7),[8](https://arxiv.org/html/2610.19013#bib.bib8)\], where increasing the number of training images is generally assumed to improve representation quality\.

However, as foundation models expand into domain\-specific visual settings such as remote sensing and autonomous driving, the assumption that increasing the number of unique samples always improves performance becomes less reliable\. Many real\-world visual corpora contain substantial semantic redundancy, near\-duplicate samples, and uneven informational density\. In such settings, increasing nominal dataset size may not correspond to increasing effective semantic coverage\.

In this work, we examine where existing data\-constrained scaling laws are insufficient to describe the trade\-off between unique samples and repetition in visual self\-distillation\. Through experiments on Maxar satellite imagery\[[9](https://arxiv.org/html/2610.19013#bib.bib9)\], SATLAS\[[10](https://arxiv.org/html/2610.19013#bib.bib10)\], the Waymo Open Dataset\[[11](https://arxiv.org/html/2610.19013#bib.bib11)\], nuPlan, Ego4D, ImageNet\[[12](https://arxiv.org/html/2610.19013#bib.bib12)\], and YFCC15M, we demonstrate that the interaction between unique sample count and repetition density is highly dataset\-dependent\.

Most strikingly, we find that a random subset containing only1\.5%1\.5\\%of the Full Maxar pool improves average downstream kNN performance by 3\.11 percentage points compared to pretraining on the full 50M\-image pool under an identical budget of 200M image presentations\. Rather than rejecting scaling laws altogether, our findings suggest that their assumptions do not fully capture the observed smaller\-subset advantage\.

To study this phenomenon, we formalize three core variables:

1. 1\.Unique data size \(UU\):The number of distinct samples in the selected pretraining pool\.
2. 2\.Repetitions or Density \(RR\):The average number of exposures per encountered unique sample during pretraining\.
3. 3\.Compute Budget \(CC\):The total number of source\-image presentations, used as a compute proxy within a fixed model and training recipe\.

To account for partial passes, we define encountered support asUseen=min⁡\(C,U\)U\_\{\\mathrm\{seen\}\}=\\min\(C,U\)and repetition asR=C/UseenR=C/U\_\{\\mathrm\{seen\}\}, givingC=Useen​RC=U\_\{\\mathrm\{seen\}\}R\. We use the terms*repetition*,*density*, and*repetition density*interchangeably forRRthroughout the paper\.

Compute\-only scaling laws collapse unique support and repetition into the single exposure termCC\. Data\-constrained laws already distinguish these quantities, but their assumptions about the value of repeated data do not capture every observed trade\-off in visual self\-distillation\.

While prior work in NLP\[[13](https://arxiv.org/html/2610.19013#bib.bib13)\]shows that repeated exposure to the same tokens leads to diminishing returns, we observe different behavior in some visual self\-distillation settings\. Unlike autoregressive language modeling, repeated exposures in visual self\-distillation are not equivalent to replaying identical training targets\. Each image exposure is transformed through stochastic augmentations, while the teacher network evolves continuously through EMA updates\. As a result, repeated views of the same image correspond to dynamically changing supervisory signals rather than static token repetition\. These differences motivate our study, but do not establish the mechanism behind the subset advantage\.

Our contributions are threefold\. First, we provide empirical evidence that repeatedly training on random subsets can outperform training on substantially larger sets of unique samples under fixed compute budgets\. Second, we compare existing scaling laws with a density\-interaction extension of a Chinchilla\-style compute\-only law that captures interactions between unique images and repetitions\. This formulation improves held\-out prediction on some datasets, although its benefits are not consistent across all seven\. Finally, we examine how the subset advantage depends on model size, self\-distillation method, subset selection, and augmentation\.

Our results suggest that scaling behavior in image self\-distillation depends not only on dataset size, but also on how the training budget is allocated between unique support and repeated exposure\.

## 2Related Work

Self\-Supervised Pretraining and Distillation:The success of large\-scale pretraining in NLP\[[1](https://arxiv.org/html/2610.19013#bib.bib1),[2](https://arxiv.org/html/2610.19013#bib.bib2)\]has motivated the development of vision foundation models trained using self\-supervised learning\[[5](https://arxiv.org/html/2610.19013#bib.bib5),[6](https://arxiv.org/html/2610.19013#bib.bib6),[7](https://arxiv.org/html/2610.19013#bib.bib7),[8](https://arxiv.org/html/2610.19013#bib.bib8)\]\. Existing approaches are broadly divided into Masked Image Modeling \(MIM\) and self\-distillation methods\. MIM approaches, such as MAE\[[5](https://arxiv.org/html/2610.19013#bib.bib5)\], reconstruct masked image regions or latent tokens, while self\-distillation approaches learn representations by matching a student network to the predictions of a teacher network\[[14](https://arxiv.org/html/2610.19013#bib.bib14)\]\. Our work focuses on iBOT\[[15](https://arxiv.org/html/2610.19013#bib.bib15)\], which combines masked prediction with online self\-distillation through an EMA\-updated teacher\. This training paradigm is particularly relevant to our study because repeated exposures to the same image do not correspond to identical supervisory signals\.

Unlike autoregressive language modeling, repeated exposures in visual self\-distillation are not equivalent to replaying identical training targets\. Each exposure corresponds to a different stochastic augmentation, while the teacher network evolves continuously through EMA updates\. As a result, repeated views of the same image provide dynamically changing supervisory signals rather than static token reuse\. This distinction may partially explain why repetition behaves differently in visual SSL compared to data\-constrained language modeling\.

Scaling Laws and Compute\-Optimal Training:Scaling laws have emerged as a central framework for understanding the relationship between model performance, dataset size, and compute\[[4](https://arxiv.org/html/2610.19013#bib.bib4),[3](https://arxiv.org/html/2610.19013#bib.bib3)\]\. In language modeling, compute\-optimal training laws suggest that performance depends jointly on model size and the number of unique training tokens, while repeated exposure to the same data rapidly yields diminishing returns\[[13](https://arxiv.org/html/2610.19013#bib.bib13)\]\. Similar assumptions are commonly adopted in visual representation learning, where larger image collections are generally expected to improve downstream performance\.

Recent work has also begun exploring scaling behavior under data filtering and constrained compute budgets\[[16](https://arxiv.org/html/2610.19013#bib.bib16),[17](https://arxiv.org/html/2610.19013#bib.bib17),[18](https://arxiv.org/html/2610.19013#bib.bib18)\]\. These studies suggest that data quality, filtering strategy, and subset selection interact strongly with training compute, and that curated subsets can outperform substantially larger unfiltered corpora in certain regimes\. However, existing formulations still primarily treat total data exposure as a single scaling axis\.

Our work revisits this assumption in the context of visual self\-distillation\. Traditional scaling laws implicitly collapse unique sample count and repetition into a single exposure term, treating repeated exposures as approximately interchangeable with additional unique data under fixed compute budgets\. In contrast, our results suggest that repetition density and unique sample count play fundamentally different roles in highly redundant visual domains\.

Data Curation, Filtering, and Pruning:Modern large\-scale pretraining increasingly relies on data curation and filtering to improve training efficiency and dataset quality\[[7](https://arxiv.org/html/2610.19013#bib.bib7),[19](https://arxiv.org/html/2610.19013#bib.bib19),[20](https://arxiv.org/html/2610.19013#bib.bib20)\]\. Prior work has shown that carefully selected subsets can outperform substantially larger unfiltered datasets, particularly when large\-scale corpora contain duplicates, noisy samples, or low\-information examples\[[21](https://arxiv.org/html/2610.19013#bib.bib21)\]\. Recent studies further demonstrate that the utility of curated subsets depends strongly on the available compute budget\[[16](https://arxiv.org/html/2610.19013#bib.bib16)\]\.

Our work is closely related to this line of research, but differs in its primary objective\. Unlike prior work on data pruning or filtering, our goal is not solely to identify smaller high\-quality subsets, but to study how repetition density and unique sample count interact under fixed compute budgets\. In particular, we show that scaling behavior itself changes across datasets with different redundancy structures\. While ImageNet largely follows conventional scaling behavior, highly redundant domains such as Maxar exhibit regimes in which repeatedly training on curated subsets can outperform training on substantially larger collections of unique samples\.

Redundancy, Duplicate Data, and Effective Dataset Size:Several recent works argue that nominal dataset size can substantially overestimate the effective information content of a corpus due to duplicates, semantic overlap, and low\-information samples\[[20](https://arxiv.org/html/2610.19013#bib.bib20),[21](https://arxiv.org/html/2610.19013#bib.bib21)\]\. These findings suggest that representation quality depends not only on dataset scale, but also on the distribution and diversity of samples\.

Our work extends this perspective to self\-distillation learning by analyzing how semantic redundancy interacts with repetition during training\. We hypothesize that performance degradation in large, highly redundant datasets does not arise solely from the presence of low\-information or noisy samples, but also from their*joint interaction within a fixed subset*\[[22](https://arxiv.org/html/2610.19013#bib.bib22)\]\. In particular, certain samples may become mutually detrimental when co\-occurring in the same training set under self\-distillation objectives, leading to amplified redundancy effects and less effective representation learning under fixed compute budgets\. Sampling can partially mitigate this effect by decorrelating such harmful co\-occurrences, thereby reducing their collective negative impact on optimization\. To quantify effective semantic coverage, we measure Vendi scores\[[23](https://arxiv.org/html/2610.19013#bib.bib23)\]over DINOv2 embeddings and analyze how diversity structure influences downstream performance across different repetition regimes\.

## 3Experimental Design

### 3\.1Pretraining Datasets

We pretrain independently on seven datasets covering four visual domains\.

Remote sensing:Maxar and SATLAS contain satellite imagery, with Full experimental pools of 50,000,000 and 10,969,092 images, respectively\. For Maxar, the original images were divided into non\-overlapping500×500500\\times 500\-pixel tiles, producing 51,197,237 tiles; our Full training pool uses 50M of these\.

Driving:Waymo and nuPlan contain 5,806,392 and 25,481,542 images, respectively\. Both provide driving scenes with temporal redundancy from nearby video frames\.

Egocentric video:Ego4D provides 38,387,515 frames depicting everyday activities from a first\-person perspective\.

Natural images:ImageNet contains 1,281,167 curated, object\-centered training images, while our YFCC15M pool contains 14,825,236 images drawn from YFCC\.

Throughout the paper, “Full” refers to the complete experimental training pool, which may differ from the complete source dataset\.

![Refer to caption](https://arxiv.org/html/2610.19013v1/figure1_pretraining_samples.png)Figure 1:Samples from pretraining datasets
### 3\.2Sampling Strategies

The experimental design serves two purposes\. First, we vary the unique\-data supportUUusing uniformly sampled random subsets to study its interaction with repetition at fixed training budgets\. We define encountered support asUseen=min⁡\(C,U\)U\_\{\\mathrm\{seen\}\}=\\min\(C,U\)and repetition asR=C/UseenR=C/U\_\{\\mathrm\{seen\}\}\. Second, we compare different subset\-selection methods at matched subset sizes and training budgets to examine whether the observed subset advantage depends on the selection strategy\.

Random samplingselects images uniformly from the parent dataset without explicitly favoring image quality or semantic coverage\. The main scaling\-law experiments use only eligible random subsets and Full\. For Maxar, these contain 50M, 25M, 12\.5M, 8\.045397M, 2M, 0\.75M, and 0\.4M images\. Audited unstable checkpoints and the ImageNet random subsets containing 1,241,131 and 1,201,094 images are excluded from both fitting and evaluation at every budget and seed\.

Hierarchical samplingconstructs subsets using representation\-space clusters\. We extract \[CLS\] embeddings with a pretrained DINOv2 model\[[7](https://arxiv.org/html/2610.19013#bib.bib7)\], then apply hierarchicalkk\-means clustering across four levels withk∈\{50​k,10​k,5​k,1​k\}k\\in\\\{50\\text\{k\},10\\text\{k\},5\\text\{k\},1\\text\{k\}\\\}\. Balanced sampling across clusters reduces the dominance of densely populated clusters\.

We also compare random and hierarchical sampling with DASSOT, a graph\-matching approach to balanced data selection\[[24](https://arxiv.org/html/2610.19013#bib.bib24)\], and quality\-based selection using the Natural Image Quality Evaluator \(NIQE\)\[[25](https://arxiv.org/html/2610.19013#bib.bib25)\]\. This comparison uses 0\.75M\-image Maxar subsets at matched training budgets and is separate from scaling\-law fitting\.

### 3\.3Model Pretraining

We pretrain iBOT with a ViT\-B/16 backbone following the standard protocol\[[15](https://arxiv.org/html/2610.19013#bib.bib15)\], using the same multi\-resolution augmentations as the original implementation\. Optimization is performed with AdamW using a per\-GPU batch size of200200across88GPUs, resulting in a global batch size ofB=1600B=1600\. All experiments are run in distributed mode with FlashAttention\-2\[[26](https://arxiv.org/html/2610.19013#bib.bib26)\]for efficiency\.

We perform pretraining across compute budgetsC∈\{10​M,20​M,30​M,40​M,50​M,100​M,200​M,300​M,400​M\}C\\in\\\{10\\text\{M\},20\\text\{M\},30\\text\{M\},40\\text\{M\},50\\text\{M\},100\\text\{M\},200\\text\{M\},\\\\ 300\\text\{M\},400\\text\{M\}\\\}\. Figure[2](https://arxiv.org/html/2610.19013#S3.F2)illustrates the experimental grid and evaluates whether subset selection meaningfully affects pretrained representation quality\. We use two sampling strategies: random uniform sampling and hierarchical sampling\.

Given batch sizeB=1600B=1600, the number of optimization steps isS=C/BS=C/B\(e\.g\.,1010M exposures≈6,250\\approx 6\{,\}250steps,5050M≈31,250\\approx 31\{,\}250steps\)\.

Figure 2:Pretraining grid across datasets\. Each point corresponds to a run with unique data sizeUUand compute budgetCC\(both in millions\)\. Colors denote Maxar, Waymo, and ImageNet\. Markers indicate sampling strategy: full data, random subsets, and hierarchical subsets\. Diagonal lines show constant repetitionRR, highlighting the trade\-off between unique data and reuse\. The fire icons indicate non\-stable pre\-trainings\.
### 3\.4Downstream Evaluation

We evaluate frozen representations using kNN classification and logistic regression on normalized final\-layer \[CLS\] embeddings\. For each pretraining group and classification task, we selectk∈\{1,5,10,30,50\}k\\in\\\{1,5,10,30,50\\\}using mean accuracy across Full\-50M reference seeds, breaking ties toward the smallestkk\. This choice remains fixed across models, budgets, and seeds\. Logistic regression uses the Cyanure solver with the saved task\-level regularization selections\.

Remote sensing:Maxar and SATLAS share six classification tasks: Brick Kiln, EuroSAT, RESISC45, UCMerced, PatternNet, and AID\.

Driving:Waymo and nuPlan share BDD100K scene and weather classification and ACDC condition classification\. Their kNN aggregates additionally include BDD100K MOT instance\- and category\-level scores\. These tracking scores have nokkand are excluded from logistic\-regression aggregates\.

General and egocentric recognition:Ego4D, ImageNet, and YFCC15M share 24 classification tasks, including ImageNet\-1K validation in both evaluation suites\. Ego4D additionally uses two EPIC\-KITCHENS\-100 noun\-recognition tasks, based on single\-frame and five\-frame pooled features\.

We average available task scores within each pretraining seed, then across available seeds, keeping kNN and logistic regression separate\. Missing scores are not replaced by zero\. The main results use kNN; logistic\-regression results are reported in Appendix[D](https://arxiv.org/html/2610.19013#A4)\. Task details and fixedkkvalues appear in Appendix[B](https://arxiv.org/html/2610.19013#A2)\.

## 4Empirical Observations

Figure 3:Behaviour of different random subsets under the kNN evaluation protocol\. The first column fixes computeCCand plots downstream performance as a function of unique imagesUU\. The middle column fixesUUand plots performance as a function of computeCC\. The third column fixesUUand plots performance as a function of repetitionRR\. The figure shows the trade\-off between unique support and repetition for full\-data and random\-subset runs\.Figure[3](https://arxiv.org/html/2610.19013#S4.F3)summarizes the empirical data\-density behavior under kNN evaluation; the corresponding logistic\-regression curves are provided in Appendix[C](https://arxiv.org/html/2610.19013#A3)\. A central empirical question is whether data\-constrained scaling behavior from language modeling transfers to image self\-distillation\. Data\-constrained language\-model laws\[[13](https://arxiv.org/html/2610.19013#bib.bib13)\]formalize a simple fixed\-compute trade\-off: decreasing the number of unique samplesUUand increasing repetitionRRshould generally hurt performance because repeated data has diminishing value\. In those experiments, repeated data behaves close to fresh data only for a small number of epochs; the marginal value decreases rapidly after that and is nearly exhausted at large repetition\. This assumption is natural for autoregressive language modeling, but it is not obviously valid for image self\-distillation, where each repeated image is transformed by stochastic augmentations and supervised by a changing EMA teacher\.

Observation 1: the fixed\-computeU/RU/Rtrade\-off is dataset\-dependent\.At fixed compute, the three datasets show qualitatively different behavior in Figure[3](https://arxiv.org/html/2610.19013#S4.F3)\. ImageNet is support\-limited at high compute: reducingUUfrom the full dataset to roughly half the images already causes a large performance drop, while further reductions continue to hurt but with a less abrupt slope\. For example, atC=100C=100M, the full ImageNet run reaches0\.69560\.6956average kNN performance, whileU≈F/2U\\approx F/2reaches0\.65450\.6545\. Waymo is much more repetition\-tolerant: after excluding runs with known training instabilities, reducingUUby factors of44to1616changes performance only weakly at matched compute\. Maxar reverses the usual intuition: reduced\-UUsubsets can outperform the full dataset by a wide margin, with the best high\-compute kNN performance occurring aroundU≈F/50U\\approx F/50toF/100F/100rather than at full data\.

This means that the fixed\-compute trade\-off is not a universal curve\. Only Waymo is close to the intuition suggested by data\-constrained language\-model scaling laws\. ImageNet penalizes the first reductions in unique support earlier than expected, while Maxar shows that repeated smaller subsets can be better than using all available unique images\.

Observation 2: the trade\-off changes with compute\.The relationship amongCC,UU, andRRis not fixed even within a dataset\. ImageNet is relatively flat at lower compute: atC=20C=20M, several reduced\-UUsubsets are close to or above the full\-data run\. The strong support\-limited behavior appears at larger compute, especially atC=100C=100M and above\. Waymo remains comparatively flat across the tested non\-spike budgets\. Maxar shows the opposite pattern: even at low compute, moving from full data to smaller subsets can improve performance, and the advantage of smallUUbecomes clearer as compute grows\. Thus, repetition cannot be treated as a dataset\-independent correction to compute; the effective value of repetition depends jointly on the domain and the budget\.

Observation 3: more unique data can be harmful, not merely redundant\.Maxar is the clearest case\. At matched compute, full\-data training often underperforms much smaller random subsets that are repeated many times\. This is stronger than saying that repetition remains useful: it suggests that additional unique images can introduce harmful modes, domain conflict, or task\-irrelevant variation that degrades the learned representation under self\-distillation\. For instance, atC=100C=100M, full\-data Maxar runs are around0\.8950\.895–0\.8970\.897, while subsets nearU=0\.5U=0\.5M–11M reach about0\.9210\.921–0\.9220\.922\. AtC=400C=400M, full\-data Maxar falls to about0\.8860\.886, whileU=0\.4U=0\.4M remains around0\.9140\.914\. This behavior motivates treating unique support as a source of both coverage and possible harm, rather than as a monotone benefit\.

Observation 4: repetition can help, saturate, or hurt\.The effect of increasingRRis not a single diminishing\-return curve\. In ImageNet, very highRRmostly reflects lost unique support and is associated with worse high\-compute transfer\. In Waymo, very largeRRcan preserve performance over a broad range ofUU, so repetition is mostly tolerated\. In Maxar, largeRRcan actively improve kNN performance, but training for too long with large or fullUUcan also degrade kNN performance, consistent with overfitting or harmful\-mode accumulation\. This degradation is much less visible under logistic\-regression evaluation, where Maxar more often appears to saturate rather than decline; we include the corresponding logistic\-regression curves in Appendix[C](https://arxiv.org/html/2610.19013#A3)\.

We exclude runs with known training\-loss spikes from these observations, because they reflect optimization instability rather than the data\-density trade\-off\. Concretely, the Waymo full\-data run drops from0\.77140\.7714atC=100C=100M to0\.58040\.5804atC=200C=200M after a loss spike, and the Waymo6\.25%6\.25\\%random subset drops from0\.76800\.7680atC=100C=100M to0\.62840\.6284atC=200C=200M after the same kind of instability\. These checkpoints are useful diagnostics for training stability, but they are not treated as valid scaling\-law observations\.

Together, these observations motivate a density\-aware view of image self\-distillation\. The empirical object is not compute alone, and not unique data alone, but the interaction among total computeCC, unique supportUU, and repetition densityRR\. Existing language\-model heuristics are useful baselines, but their assumptions about repeated data do not consistently transfer to image self\-distillation\.

## 5Density\-Aware Scaling Law

The empirical observations above raise two questions: Can existing scaling\-law parameterizations explain the interaction among total compute, unique data, and repetition in image self\-distillation? And, by using only small\-budget runs, can scaling laws determine which pretraining subset to use at a target compute budget to maximize average downstream accuracy?

#### Fitting protocol\.

We fit every law independently for each pretraining dataset\. The target is average downstream accuracy across the corresponding frozen\-representation evaluation tasks, so the reported error measures how well a law predicts transfer performance rather than pretraining loss\. Unless stated otherwise, the main results use the kNN aggregate\. Parameters are optimized only on low\-compute runs withC≤50C\\leq 50M image views and evaluated on higher\-compute runs withC\>50C\>50M\. All laws use the same underlying quantities: total seen imagesCC, selected pool sizeUU, encountered unique imagesUseen=min⁡\(C,U\)U\_\{\\mathrm\{seen\}\}=\\min\(C,U\), induced repetitionR=C/UseenR=C/U\_\{\\mathrm\{seen\}\}\. We measureCCandUseenU\_\{\\mathrm\{seen\}\}in millions of image presentations and unique images, respectively\. RepetitionRRis dimensionless\. We use the same post\-spike exclusions described in Section[4](https://arxiv.org/html/2610.19013#S4)\. We report mean absolute error \(MAE\) separately in the fit range and in the extrapolation range\.

Figure 4:Scaling\-law fits and extrapolations under the kNN evaluation protocol\. Columns compare three existing\-law baselines—compute\-only, data\-constrained, and prescriptive—with our density\-interaction law\. These four forms test whether compute alone, diminishing returns from repetition, a repetition penalty, or a support–repetition interaction captures the observed behavior\. Rows show Maxar, nuPlan, ImageNet, and YFCC15M\. For compactness, SATLAS and Ego4D are omitted because they show similar qualitative patterns to Maxar, while Waymo shows similar observed trends to nuPlan\. Curves are fitted only on the shaded low\-compute region and evaluated on higher\-compute runs\. Fitting curves for all datasets and the logistic\-regression comparison can be found in Appendices[D](https://arxiv.org/html/2610.19013#A4)and[F](https://arxiv.org/html/2610.19013#A6)\.
#### Existing laws\.

We compare three baseline parameterizations adapted to predict downstream accuracy at fixed model size\. The first baseline is a compute\-only Chinchilla\-style power law,

y^​\(C\)=L−ACα,\\hat\{y\}\(C\)=L\-\\frac\{A\}\{C^\{\\alpha\}\},\(1\)which captures improvement with total training exposure but cannot distinguish two runs with the sameCCand different\(U,R\)\(U,R\)\. The second baseline uses a data\-constrained effective\-data parametrization:

Ueff​\(Useen,R\)=Useen​\[1\+τ⁡\(1−exp⁡\(−R−1τ\)\)\],y^​\(U,R\)=L−AUeff​\(Useen,R\)β,U\_\{\\mathrm\{eff\}\}\(U\_\{\\mathrm\{seen\}\},R\)=U\_\{\\mathrm\{seen\}\}\[1\+\\tau\\left\(1\-\\exp\\left\(\-\\frac\{R\-1\}\{\\tau\}\\right\)\\right\)\],\\qquad\\hat\{y\}\(U,R\)=L\-\\frac\{A\}\{U\_\{\\mathrm\{eff\}\}\(U\_\{\\mathrm\{seen\}\},R\)^\{\\beta\}\},\(2\)
Hereτ\\taucontrols the saturation scale of the contribution from additional passes\. This law gives additional passes diminishing value, with the rate controlled byτ\\tau\. The third baseline is an adaptation of the prescriptive law, which adds a repetition\-dependent penalty:

y^​\(C,U,R\)=L−ACα−B​\(R−1\)δUseenη,\\hat\{y\}\(C,U,R\)=L\-\\frac\{A\}\{C^\{\\alpha\}\}\-\\frac\{B\(R\-1\)^\{\\delta\}\}\{U\_\{\\mathrm\{seen\}\}^\{\\eta\}\},\(3\)
The compute\-only law gives the same prediction to all subsets at a fixed compute budget\. The data\-constrained law allows repeated exposure to help, but cannot predict declining performance at fixed unique support\. The prescriptive law can model such a decline\. However, under our parameter constraints, none of these baselines can predict that a smaller subset strictly outperforms a larger one at the same compute budget\. Their fitted predictions are also similar on several datasets; Appendices[D](https://arxiv.org/html/2610.19013#A4)and[F](https://arxiv.org/html/2610.19013#A6)examines these similarities\.

#### Density interaction\.

To capture these differences, we extend the compute\-only baseline with a joint correction for unique support and repetition\.

Perf⁡\(C,U,R\)=L−ACα−D​\[log⁡\(1\+Useen\)\]β​\[log⁡\(1\+R\)\]γ\.\\mathrm\{Perf\}\(C,U,R\)=L\-\\frac\{A\}\{C^\{\\alpha\}\}\-D\[\\log\(1\+U\_\{\\mathrm\{seen\}\}\)\]^\{\\beta\}\[\\log\(1\+R\)\]^\{\\gamma\}\.\(4\)The compute term captures improvement with total training exposure\. The interaction term adjusts the prediction according to how this exposure is divided between unique images and repetitions\. We interpret it as a phenomenological density penalty, not a first\-principles law\. One possible mechanism is that harmful, low\-information, or domain\-conflicting modes are scattered across the unique sample pool\. A smaller random subset may omit some of these modes, while repeated training may amplify the effects of those retained\. This remains a hypothesis rather than a mechanism established by our experiments\. The logarithms avoid treating nominal unique images or repetitions as linearly independent units, and the exponentsβ\\betaandγ\\gammacontrol dataset\-specific sensitivity to unique support and repetition density\. These coefficients should be interpreted jointly withDD, not as standalone mechanistic measurements\.

Table 1:kNN scaling\-law MAE in accuracy units, fitted onC≤50C\\leq 50M and evaluated onC\>50C\>50M\. Only eligible random subsets and Full are included\. Each task uses a fixedkkchosen from Full\-50M reference seeds\. Bold and underlining indicate the lowest and second\-lowest distinct unrounded held\-out errors in each dataset\.
#### Main results\.

Table[1](https://arxiv.org/html/2610.19013#S5.T1)shows that density interaction achieves the lowest held\-out MAE on Maxar, SATLAS, and Ego4D, and remains close to the best baseline on Waymo\. However, it performs worse on nuPlan, ImageNet, and YFCC15M\. Thus, allowing a smaller\-subset advantage improves prediction in some settings but does not guarantee better extrapolation\.

We next examine whether the observed subset advantage persists across model sizes, self\-distillation methods, subset\-selection strategies, and augmentation settings\.

#### Effect of model size\.

To test whether the subset advantage is specific to ViT\-B, we also train iBOT with ViT\-S and ViT\-L on Maxar\. For both additional model sizes, we evaluate four dataset sizes at training budgets from 10M to 200M image presentations\. At 100M presentations, the random 0\.75M\-image subset achieves average kNN accuracy of 0\.9207 with ViT\-S and 0\.9196 with ViT\-L, compared with 0\.8994 and 0\.8796 for full\-data training\. The subset advantage also appears under logistic\-regression evaluation\. These results show that the finding persists across the three tested model sizes\.

#### Effect of the self\-distillation method\.

To test whether the subset advantage is specific to iBOT, we also train DINOv2 with ViT\-B on Maxar\. At 100M image presentations, the random 0\.75M\-image subset achieves average kNN accuracy of 0\.9182, compared with 0\.9095 for full\-data training\. This advantage persists through 300M presentations and also appears under logistic\-regression evaluation\. These results support the finding across two self\-distillation methods, while its generalization to other self\-supervised learning methods remains untested\.

#### Effect of subset selection\.

To test whether the subset advantage depends on the selection method, we compare random sampling, hierarchical selection, DASSOT, and NIQE\-based quality selection on Maxar\. At 50M image presentations, the 0\.75M\-image subsets achieve average kNN accuracies of 0\.9202, 0\.9187, 0\.9202, and 0\.9216, respectively, compared with 0\.9015 for full\-data training\. The advantage across all four methods shows that it does not require a particular selection strategy\. However, these comparisons do not fully separate the effects of repetition, redundancy, image quality, and semantic coverage\.

#### Effect of augmentations\.

We hypothesize that unique images and repeated exposure may contribute differently to learning in visual self\-distillation\. We hypothesize that repeated exposure remains useful beyond the additional variation provided by strong augmentation\. To test this, we train iBOT on a random 0\.75M\-image Maxar subset with reduced augmentation\. We disable color jitter, grayscale, Gaussian blur, and solarization, while retaining cropping, horizontal flipping, normalization, and masking\.

Reduced augmentation lowers average kNN accuracy at 100M presentations from 0\.9251 to 0\.8910\. However, accuracy increases from 0\.8400 at 10M presentations to 0\.8969 at 400M\. This supports the hypothesis that useful repetition does not depend entirely on the removed augmentations\. The retained crops and masks still introduce variation, and without a reduced\-augmentation Full control, we cannot determine whether the subset advantage also persists\.

The statistical tests show that the accuracy differences captured by the Maxar scaling\-law analysis are not small numerical artifacts: 124 of 187 matched pairwise comparisons are significant at p < 0\.05, while the fitted law achieves low train and heldout error\. This supports the paper’s main outcome that, in high\-redundancy remote\-sensing data, scaling is governed not only by compute and unique\-image count but also by data density/repetition, with carefully chosen reduced\-U subsets often matching or exceeding full\-data training at the same compute\.

## Conclusion

We studied empirical scaling behavior for image self\-distillation when total computeCC, unique supportUU, and repetitionR=C/UR=C/Uare varied independently\. The main finding is that data density is not a minor implementation detail: reduced unique support can hurt early on ImageNet, remain nearly neutral on Waymo, and substantially improve Maxar at matched compute\. Existing compute\-only and data\-constrained laws do not consistently capture these behaviors, while a density\-interaction law gives a better empirical description in the tested extrapolation range\.

## References

- \[1\]Radford, A\., J\. Wu, R\. Child, et al\.Language models are unsupervised multitask learners\.*OpenAI blog*, 1\(8\):9, 2019\.
- \[2\]Brown, T\. B\., B\. Mann, N\. Ryder, et al\.Language models are few\-shot learners\.In H\. Larochelle, M\. Ranzato, R\. Hadsell, M\. Balcan, H\. Lin, eds\.,*Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6\-12, 2020, virtual*\. 2020\.
- \[3\]Hoffmann, J\., S\. Borgeaud, A\. Mensch, et al\.Training compute\-optimal large language models\.*CoRR*, abs/2203\.15556, 2022\.
- \[4\]Kaplan, J\., S\. McCandlish, T\. Henighan, et al\.Scaling Laws for Neural Language Models\.*arXiv e\-prints*, arXiv:2001\.08361, 2020\.
- \[5\]He, K\., X\. Chen, S\. Xie, et al\.Masked autoencoders are scalable vision learners\.In*IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18\-24, 2022*, pages 15979–15988\. IEEE, 2022\.
- \[6\]Wang, W\., H\. Bao, L\. Dong, et al\.Image as a foreign language: Beit pretraining for vision and vision\-language tasks\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 19175–19186\. 2023\.
- \[7\]Oquab, M\., T\. Darcet, T\. Moutakanni, et al\.Dinov2: Learning robust visual features without supervision\.*Trans\. Mach\. Learn\. Res\.*, 2024, 2024\.
- \[8\]Siméoni, O\., H\. V\. Vo, M\. Seitzer, et al\.DINOv3, 2025\.
- \[9\]MaxarTechnologies\.Open data program\.[https://registry\.opendata\.aws/maxar\-open\-data](https://registry.opendata.aws/maxar-open-data), 2022\.Accessed: 2022 Apr 1\.
- \[10\]Bastani, F\., P\. Wolters, R\. Gupta, et al\.Satlaspretrain: A large\-scale dataset for remote sensing image understanding\.In*IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1\-6, 2023*, pages 16726–16736\. IEEE, 2023\.
- \[11\]Sun, P\., H\. Kretzschmar, X\. Dotiwalla, et al\.Scalability in perception for autonomous driving: Waymo open dataset\.*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2020\.
- \[12\]Deng, J\., W\. Dong, R\. Socher, et al\.Imagenet: A large\-scale hierarchical image database\.*CVPR*, 2009\.
- \[13\]Muennighoff, N\., A\. M\. Rush, B\. Barak, et al\.Scaling data\-constrained language models\.In A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, S\. Levine, eds\.,*Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023*\. 2023\.
- \[14\]Caron, M\., H\. Touvron, I\. Misra, et al\.Emerging properties in self\-supervised vision transformers\.In*2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10\-17, 2021*, pages 9630–9640\. IEEE, 2021\.
- \[15\]Zhou, J\., C\. Wei, H\. Wang, et al\.ibot: Image BERT pre\-training with online tokenizer\.*CoRR*, abs/2111\.07832, 2021\.
- \[16\]Goyal, S\., P\. Maini, Z\. C\. Lipton, et al\.Scaling laws for data filtering \- data curation cannot be compute agnostic\.In*IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16\-22, 2024*, pages 22702–22711\. IEEE, 2024\.
- \[17\]Wiegand, G\.\-H\., L\. Raichle, R\. Staedeli, et al\.Is more data worth the cost? dataset scaling laws in a tiny attention\-only decoder, 2026\.DATA\-FM Workshop @ ICLR 2026\.
- \[18\]Ye, J\., V\. Feldman, K\. Talwar\.Cram less to fit more: Training data pruning improves memorization of facts, 2026\.DATA\-FM Workshop @ ICLR 2026\.
- \[19\]Gadre, S\. Y\., G\. Ilharco, A\. Fang, et al\.Datacomp: In search of the next generation of multimodal datasets\.In*Advances in Neural Information Processing Systems*\. 2023\.
- \[20\]Abbas, A\., K\. Tirumala, D\. Simig, et al\.Semdedup: Data\-efficient learning at web\-scale through semantic deduplication\.*CoRR*, abs/2303\.09540, 2023\.
- \[21\]Sorscher, B\., R\. Geirhos, S\. Shekhar, et al\.Beyond neural scaling laws: beating power law scaling via data pruning\.In*Advances in Neural Information Processing Systems*, vol\. 35, pages 19523–19536\. 2022\.
- \[22\]Hu, Y\., P\. Hu, H\. Zhao, et al\.Most influential subset selection: Challenges, promises, and beyond\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, vol\. 37\. 2024\.
- \[23\]Friedman, D\., A\. B\. Dieng\.The vendi score: A diversity evaluation metric for machine learning\.*Transactions on Machine Learning Research*, 2023\.
- \[24\]Van Assel, H\., R\. Balestriero\.A graph matching approach to balanced data sub\-sampling for self\-supervised learning\.In*NeurIPS 2024 Workshop: Self\-Supervised Learning\-Theory and Practice*\.
- \[25\]Mittal, A\., R\. Soundararajan, A\. C\. Bovik\.Making a “completely blind” image quality analyzer\.*IEEE Signal Processing Letters*, 20\(3\):209–212, 2013\.
- \[26\]Dao, T\.Flashattention\-2: Faster attention with better parallelism and work partitioning\.*arXiv preprint arXiv:2307\.08691*, 2023\.
- \[27\]Lee, J\., N\. R\. Brooks, F\. Tajwar, et al\.Scalable deep learning to identify brick kilns and aid regulatory capacity\.*Proceedings of the National Academy of Sciences*, 118\(17\):e2018863118, 2021\.
- \[28\]Helber, P\., B\. Bischke, A\. Dengel, et al\.Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification\.*IEEE J\. Sel\. Top\. Appl\. Earth Obs\. Remote\. Sens\.*, 12\(7\):2217–2226, 2019\.
- \[29\]Cheng, G\., J\. Han, X\. Lu\.Remote sensing image scene classification: Benchmark and state of the art\.*Proceedings of the IEEE*, 105\(10\):1865–1883, 2017\.
- \[30\]Yang, Y\., S\. D\. Newsam\.Bag\-of\-visual\-words and spatial extensions for land\-use classification\.In D\. Agrawal, P\. Zhang, A\. E\. Abbadi, M\. F\. Mokbel, eds\.,*18th ACM SIGSPATIAL International Symposium on Advances in Geographic Information Systems, ACM\-GIS 2010, November 3\-5, 2010, San Jose, CA, USA, Proceedings*, pages 270–279\. ACM, 2010\.
- \[31\]Zhou, W\., S\. Newsam, C\. Li, et al\.PatternNet: A benchmark dataset for performance evaluation of remote sensing image retrieval\.*ISPRS Journal of Photogrammetry and Remote Sensing*, 145:197–209, 2018\.
- \[32\]Xia, G\.\-S\., J\. Hu, F\. Hu, et al\.AID: A benchmark data set for performance evaluation of aerial scene classification\.*IEEE Transactions on Geoscience and Remote Sensing*, 55\(7\):3965–3981, 2017\.
- \[33\]Yu, F\., H\. Chen, X\. Wang, et al\.Bdd100k: A diverse driving dataset for heterogeneous multitask learning\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 2636–2645\. 2020\.
- \[34\]Sakaridis, C\., D\. Dai, L\. Van Gool\.ACDC: The adverse conditions dataset with correspondences for semantic driving scene understanding\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*, pages 10765–10775\. 2021\.
- \[35\]Krause, J\., M\. Stark, J\. Deng, et al\.3d object representations for fine\-grained categorization\.In*Proceedings of the IEEE International Conference on Computer Vision Workshops*, pages 554–561\. 2013\.
- \[36\]Krizhevsky, A\.Learning multiple layers of features from tiny images\.Tech\. rep\., University of Toronto, 2009\.
- \[37\]Wah, C\., S\. Branson, P\. Welinder, et al\.The caltech\-ucsd birds\-200\-2011 dataset\.Tech\. Rep\. CNS\-TR\-2011\-001, California Institute of Technology, 2011\.
- \[38\]Cimpoi, M\., S\. Maji, I\. Kokkinos, et al\.Describing textures in the wild\.In*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pages 3606–3613\. 2014\.
- \[39\]Goodfellow, I\. J\., D\. Erhan, P\. L\. Carrier, et al\.Challenges in representation learning: A report on three machine learning contests\.*Neural Networks*, 64:59–63, 2015\.
- \[40\]Maji, S\., J\. Kannala, E\. Rahtu, et al\.Fine\-grained visual classification of aircraft\.Tech\. rep\., 2013\.
- \[41\]Nilsback, M\.\-E\., A\. Zisserman\.Automated flower classification over a large number of classes\.In*Indian Conference on Computer Vision, Graphics and Image Processing*\. 2008\.
- \[42\]Bossard, L\., M\. Guillaumin, L\. Van Gool\.Food\-101 – mining discriminative components with random forests\.In*European Conference on Computer Vision*\. 2014\.
- \[43\]Stallkamp, J\., M\. Schlipsing, J\. Salmen, et al\.The german traffic sign recognition benchmark: A multi\-class classification competition\.In*Proceedings of the 2011 International Joint Conference on Neural Networks*, pages 1453–1460\. 2011\.
- \[44\]LeCun, Y\., L\. Bottou, Y\. Bengio, et al\.Gradient\-based learning applied to document recognition\.*Proceedings of the IEEE*, 86\(11\):2278–2324, 1998\.
- \[45\]Veeling, B\. S\., J\. Linmans, J\. Winkens, et al\.Rotation equivariant cnns for digital pathology\.In*Medical Image Computing and Computer Assisted Intervention – MICCAI 2018*, pages 210–218\. 2018\.
- \[46\]Parkhi, O\. M\., A\. Vedaldi, A\. Zisserman, et al\.Cats and dogs\.In*IEEE Conference on Computer Vision and Pattern Recognition*\. 2012\.
- \[47\]Coates, A\., H\. Lee, A\. Y\. Ng\.An analysis of single\-layer networks in unsupervised feature learning\.In*Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics*, pages 215–223\. 2011\.
- \[48\]Fei\-Fei, L\., R\. Fergus, P\. Perona\.Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories\.*Computer Vision and Image Understanding*, 106\(1\):59–70, 2007\.
- \[49\]Lomonaco, V\., D\. Maltoni\.CORe50: a new dataset and benchmark for continuous object recognition\.In*Proceedings of the 1st Annual Conference on Robot Learning*, vol\. 78 of*Proceedings of Machine Learning Research*, pages 17–26\. PMLR, 2017\.
- \[50\]Zhu, C\., F\. Xiao, A\. Alvarado, et al\.EgoObjects: A large\-scale egocentric dataset for fine\-grained object understanding\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*, pages 20110–20120\. 2023\.
- \[51\]Quattoni, A\., A\. Torralba\.Recognizing indoor scenes\.In*2009 IEEE Conference on Computer Vision and Pattern Recognition*, pages 413–420\. 2009\.
- \[52\]Damen, D\., H\. Doughty, G\. M\. Farinella, et al\.Rescaling egocentric vision: Collection, pipeline and challenges for EPIC\-KITCHENS\-100\.*International Journal of Computer Vision*, 130\(1\):33–55, 2022\.

## Appendix APretraining Dataset Vendi Scores

Table 2:Vendi scores by dataset, sampling strategy, and target subset percentage\.Note: Maxar uses the nearest available subset size to each target percentage based on 50M training images\. Waymo and ImageNet use exact available percentage matches; missing entries are shown as\-\-\.

## Appendix BDownstream Tasks

Maxar and SATLAS are evaluated on six remote\-sensing classification tasks \(Table[3](https://arxiv.org/html/2610.19013#A2.T3)\)\. Waymo and nuPlan use three driving classification tasks, with two additional BDD100K tracking scores included in their kNN aggregates \(Table[4](https://arxiv.org/html/2610.19013#A2.T4)\)\. Ego4D, ImageNet, and YFCC15M share 24 classification tasks; Ego4D additionally uses EPIC\-KITCHENS\-100 noun recognition with single\-frame and five\-frame pooled features \(Table[5](https://arxiv.org/html/2610.19013#A2.T5)\)\.

For each pretraining group and classification task,k∈\{1,5,10,30,50\}k\\in\\\{1,5,10,30,50\\\}maximizes mean accuracy across Full\-50M reference seeds; ties select the smallestkk\. This choice remains fixed across models, budgets, and seeds\. Tracking scores have nokk\. We aggregate kNN and logistic\-regression results separately, averaging available task scores within each seed and then across seeds\.

Table 3:Remote\-sensing downstream tasks and fixed referencekkvalues\.Table 4:Driving downstream tasks and fixed referencekkvalues\. Tracking scores enter only the kNN aggregate; their sample counts are unspecified\.Table 5:General and egocentric downstream tasks\. Fixed referencekkvalues are shown for Ego4D \(Ego\), ImageNet \(IN\), and YFCC15M \(YFCC\)\. A dash indicates that the task is not evaluated for that group\.TaskTrainEvalClassesEgokkINkkYFCCkkImageNet\-1K\[[12](https://arxiv.org/html/2610.19013#bib.bib12)\]1,281,16750,0001,000301030Stanford Cars\[[35](https://arxiv.org/html/2610.19013#bib.bib35)\]8,1448,041196111CIFAR\-100\[[36](https://arxiv.org/html/2610.19013#bib.bib36)\]50,00010,000100101030CIFAR\-10\[[36](https://arxiv.org/html/2610.19013#bib.bib36)\]50,00010,00010101010CUB\-200\[[37](https://arxiv.org/html/2610.19013#bib.bib37)\]5,9945,794200303030DTD\[[38](https://arxiv.org/html/2610.19013#bib.bib38)\]1,8801,8804710110FER\-2013\[[39](https://arxiv.org/html/2610.19013#bib.bib39)\]28,7097,1787111FGVC\-Aircraft\[[40](https://arxiv.org/html/2610.19013#bib.bib40)\]3,3343,333100111Flowers\-102\[[41](https://arxiv.org/html/2610.19013#bib.bib41)\]1,0201,020102111Food\-101\[[42](https://arxiv.org/html/2610.19013#bib.bib42)\]75,75025,250101503050GTSRB\[[43](https://arxiv.org/html/2610.19013#bib.bib43)\]26,64012,63043503050MNIST\[[44](https://arxiv.org/html/2610.19013#bib.bib44)\]60,00010,0001010510PCAM\[[45](https://arxiv.org/html/2610.19013#bib.bib45)\]262,14432,7682505050Oxford\-IIIT Pets\[[46](https://arxiv.org/html/2610.19013#bib.bib46)\]3,6803,66937103030STL\-10\[[47](https://arxiv.org/html/2610.19013#bib.bib47)\]5,0008,0001010510Caltech\-101\[[48](https://arxiv.org/html/2610.19013#bib.bib48)\]3,0305,64710110101RESISC45\[[29](https://arxiv.org/html/2610.19013#bib.bib29)\]18,9006,300451510UCMerced\[[30](https://arxiv.org/html/2610.19013#bib.bib30)\]1,26042021111EuroSAT\[[28](https://arxiv.org/html/2610.19013#bib.bib28)\]16,2005,40010155Brick Kiln\[[27](https://arxiv.org/html/2610.19013#bib.bib27)\]58,8907,3612555CORe50 identity\[[49](https://arxiv.org/html/2610.19013#bib.bib49)\]119,89444,97250105050EgoObjects category\[[50](https://arxiv.org/html/2610.19013#bib.bib50)\]447,735164,240638505050EgoObjects identity\[[50](https://arxiv.org/html/2610.19013#bib.bib50)\]27,690124,9563,510111MIT Indoor\-67\[[51](https://arxiv.org/html/2610.19013#bib.bib51)\]5,3601,34067301010EPIC noun \(frame\)\[[52](https://arxiv.org/html/2610.19013#bib.bib52)\]67,2179,66830010––EPIC noun \(pool\)\[[52](https://arxiv.org/html/2610.19013#bib.bib52)\]67,2179,6683005––
EgoObjects counts refer to object crops\. Category classification uses a 638\-label vocabulary, with 352 categories observed in training and 275 in evaluation\. Identity classification uses capture\-disjoint reference and query crops within the official test subset; Train denotes the reference bank\. EPIC\-KITCHENS\-100 counts refer to action segments, with evaluation on public validation\. Its 300\-label vocabulary contains 289 labels observed in training; all validation segments are retained\.

## Appendix CLogistic\-regression density curves

Figure 5:Unique\-support and repetition trade\-offs under logistic\-regression evaluation\. Rows correspond to Maxar, SATLAS, Waymo, nuPlan, Ego4D, ImageNet, and YFCC15M\. Columns show mean downstream accuracy as unique imagesUUvary at fixed computeCC, compute varies at fixedUU, and repetitionR=C/min⁡\(C,U\)R=C/\\min\(C,U\)varies at fixedUU, respectively\. Points represent random\-subset and Full\-data runs; lines connect observations\. Colors indicate compute in the left column and unique support in the middle and right columns, which share a color key\. Horizontal axes are logarithmic; image counts are in millions\.
## Appendix DLogistic\-regression scaling\-law results

Figure 6:Scaling\-law fits and extrapolations under the logistic\-regression evaluation protocol for all seven datasets\. Columns compare the compute\-only Chinchilla\-style law, the data\-constrained effective\-data law, the prescriptive scaling law, and our density\-interaction law\. Rows correspond to Maxar, SATLAS, Waymo, nuPlan, Ego4D, ImageNet, and YFCC15M\. Curves are fitted only in the shaded low\-compute region \(C≤50C\\leq 50M\) and evaluated on higher\-compute runs \(C\>50C\>50M\)\. Each panel shows the fitted equation and held\-out MAE in accuracy units\. Parameters are fitted separately from the kNN evaluation\.
## Appendix Ek\-NN density curves

Figure 7:Unique\-support and repetition trade\-offs under kNN evaluation\. Rows correspond to Maxar, SATLAS, Waymo, nuPlan, Ego4D, ImageNet, and YFCC15M\. The left column fixes computeCCand varies unique imagesUU; the middle column fixesUUand variesCC; the right column fixesUUand varies repetitionR=C/min⁡\(C,U\)R=C/\\min\(C,U\)\. Points show mean downstream accuracy for random\-subset and Full\-data runs, with lines connecting observations\. Colors indicate compute in the left column and unique support in the middle and right columns, which share a color key\. For each pretraining group and downstream task,kkis selected from the Full\-data 50M reference runs and held fixed across evaluated models\. Horizontal axes are logarithmic; image counts are in millions\.
## Appendix Fk\-NN scaling\-law results

Figure 8:Scaling\-law fits and extrapolations under the kNN evaluation protocol for all seven datasets\. Columns compare the compute\-only Chinchilla\-style law, the data\-constrained effective\-data law, the prescriptive scaling law, and our density\-interaction law\. Rows correspond to Maxar, SATLAS, Waymo, nuPlan, Ego4D, ImageNet, and YFCC15M\. Curves are fitted only in the shaded low\-compute region \(C≤50C\\leq 50M\) and evaluated on higher\-compute runs \(C\>50C\>50M\)\. Each panel shows the fitted equation and held\-out MAE in accuracy units\.Table 6:Logistic\-regression scaling\-law comparison\. Each law is fitted on runs withC≤50C\\leq 50M and evaluated on runs withC\>50C\>50M\. Lower MAE is better\. We useℓU=log⁡\(1\+U\)\\ell\_\{U\}=\\log\(1\+U\)andℓR=log⁡\(1\+R\)\\ell\_\{R\}=\\log\(1\+R\)\.
## Appendix GScaling\-law ablation plots

Figure 9:kNN ablation plots for the density\-interaction law\. Rows compare the proposed interaction law against unique\-only, repeat\-only, separate\-additive, raw\-UU, and raw\-RRvariants\.Figure 10:Logistic\-regression ablation plots for the density\-interaction law\.
## Appendix HFitted density\-interaction coefficients

Table 7:Fitted coefficients for the density\-interaction law,Perf=L−A/Cα−D​\[log⁡\(1\+U\)\]β​\[log⁡\(1\+R\)\]γ\\mathrm\{Perf\}=L\-A/C^\{\\alpha\}\-D\[\\log\(1\+U\)\]^\{\\beta\}\[\\log\(1\+R\)\]^\{\\gamma\}\. Coefficients are fitted independently per dataset and evaluation protocol\.
## Appendix ISubset Selection

Figure 11:Maxar matched\-pair comparison between hierarchical\-clustering selection and random selection\. Each line fixes the subset sizeUUand plots the downstream\-performance difference, hierarchical minus random, as compute increases\.