Understanding Context Sampling in TabPFN on Small Tabular Datasets

# Understanding Context Sampling in TabPFN on Small Tabular Datasets
Source: [https://arxiv.org/html/2607.26628](https://arxiv.org/html/2607.26628)
###### Abstract

TabPFN classifies by in\-context learning: it conditions on a set of labeled training rows \(the context, or prototypes\) and predicts test labels without gradient updates\. On small tabular datasets, where TabPFN is designed to be used, the practitioner still chooses how large that context is and which rows go into it\. We ask how much those choices matter\. Using repeated random sub\-sampling on 15 small OpenML datasets, we study three linked questions: \(i\) how variable predictions are across different random contexts of the same size, and whether larger contexts are more stable; \(ii\) whether accuracy tracks how well a sampled context preserves the full training distribution; and \(iii\) whether more expensive selection \(K\-Means, farthest\-point sampling\) buys anything over uniform random selection once the context is representative\. We find that larger contexts are both more accurate and markedly more stable \(the coefficient of variation of AUC falls from roughly 6–18% atk=16k\{=\}16to 1–4% at the largestkkon the datasets with room to improve\)\. Accuracy does correlate with how well a random context preserves the training distribution, but a controlled experiment overturns the causal reading: when we construct contexts that match the feature means closely, accuracy drops substantially \(by up to 0\.5 AUC\), because that construction also destroys diversity\. Controlled experiments and a mixed\-effects analysis point to diversity and coverage, rather than feature\-mean matching, as the factor that predicts accuracy \(diversityβ=\+0\.23\\beta=\+0\.23,p=3×10−12p=3\\times 10^\{\-12\}; feature\-mean shiftβ=−0\.01\\beta=\-0\.01,p=0\.71p=0\.71once diversity is included\)\. This also explains why K\-Means and farthest\-point selection, which are diverse but not distribution\-preserving, match uniform random while costing two to three orders of magnitude more\. Random sampling works because it covers the feature space in expectation, not because it matches its distribution\. We report effect sizes and confidence intervals throughout\.

Keywords:in\-context learning, TabPFN, prototype sampling, context diversity, coverage, context stability, small data

## 1Introduction

Prototype selection is a fundamental inference\-time decision in TabPFN\[[1](https://arxiv.org/html/2607.26628#bib.bib1)\], a prior\-data fitted network\[[2](https://arxiv.org/html/2607.26628#bib.bib2)\]: the model conditions on a chosen set of labeled rows \(the context\) and predicts in a single forward pass, the tabular instance of the in\-context learning paradigm popularized by large language models\[[3](https://arxiv.org/html/2607.26628#bib.bib3),[4](https://arxiv.org/html/2607.26628#bib.bib4)\]\. Prototype and instance selection have a long history in nearest\-neighbor learning\[[5](https://arxiv.org/html/2607.26628#bib.bib5)\], and sophisticated selection methods \(clustering, coverage heuristics\) have been proposed; yet it remains unclear whether their gains come from the selection*algorithm*or simply from selecting a subset that is representative of the training data\. We investigate this on small tabular datasets, the regime TabPFN targets, where we can afford many repeated draws and isolate the effect of the context from confounds of scale\.

We study three connected questions, on 15 small OpenML datasets:

H1 \(Stability\)\.Different random contexts of the same sizekkyield different predictions; the spread of accuracy across random draws shrinks askkgrows\.

H2 \(What makes a context good\)\.We ask whether accuracy is driven by how well the context preserves the training distribution \(representativeness\) or by how well it covers the feature space \(diversity\)\. A correlation initially suggests representativeness; a controlled experiment shows the causal factor is diversity\.

H3 \(Selection cost\)\.Expensive selection methods \(K\-Means, FPS\) add little accuracy over uniform random selection despite far higher cost\.

These are mechanistic questions about*how*TabPFN responds to its context, and together they form a single account: context size controls reliability \(H1\), diversity and coverage \(not distribution matching\) explain which contexts succeed \(H2\), and this explains why cheap random sampling, which covers the space in expectation, is hard to beat \(H3\)\. We answer them with repeated random sub\-sampling,*controlled*construction of contexts with targeted geometry, and two supporting stress tests \(duplicating and dropping part of the context\)\. Our contribution is to identify context diversity, rather than representativeness or the choice of selection algorithm, as what a small\-data context needs, and to reach that conclusion through a controlled experiment that overturns the more obvious correlational story\.

## 2Related Work

In\-context learning and TabPFN\.In\-context learning, in which a model adapts to a task from examples placed in its input without weight updates, was popularized by large language models\[[3](https://arxiv.org/html/2607.26628#bib.bib3)\]and has since been surveyed extensively\[[4](https://arxiv.org/html/2607.26628#bib.bib4)\]\. Prior\-data fitted networks \(PFNs\) bring this paradigm to supervised learning by meta\-training a transformer to perform Bayesian inference in a forward pass\[[2](https://arxiv.org/html/2607.26628#bib.bib2)\]; TabPFN applies it to tabular classification and regression\[[1](https://arxiv.org/html/2607.26628#bib.bib1)\]\. Our study takes TabPFN as given and asks a downstream question its design raises but does not answer: how the in\-context set should be chosen on small data\.

Prototype and instance selection\.Choosing a small subset of training examples is a classical problem in nearest\-neighbor learning, where prototype selection methods aim to shrink the reference set while preserving accuracy\[[5](https://arxiv.org/html/2607.26628#bib.bib5)\]\. Clustering\- and coverage\-based selectors are standard tools: k\-means\+\+ seeding\[[6](https://arxiv.org/html/2607.26628#bib.bib6)\]and farthest\-point /kk\-center sampling\[[7](https://arxiv.org/html/2607.26628#bib.bib7)\]both spread selected points across the input space\. We evaluate exactly these families as context selectors for TabPFN and find that, at matched budget, they do not beat uniform random enough to justify their cost\.

Coverage, diversity, and coresets\.A parallel line of work formalizes “covering” a dataset with a small set\. Determinantal point processes model diverse subsets through a volume \(log\-determinant\) objective\[[8](https://arxiv.org/html/2607.26628#bib.bib8)\]; coreset constructions seek small weighted subsets that approximate a learning objective\[[9](https://arxiv.org/html/2607.26628#bib.bib9)\], and coverage\-driven selection has been used directly for active learning\[[10](https://arxiv.org/html/2607.26628#bib.bib10)\]\. These motivate the geometric diversity/coverage descriptors we use, and our controlled experiments provide evidence that, for TabPFN on small data, such coverage rather than distributional fidelity is what tracks accuracy\.

Data valuation and compression\.Related efforts ask which data are valuable or how to compress a dataset: data\-valuation methods score individual examples by their marginal contribution\[[12](https://arxiv.org/html/2607.26628#bib.bib12)\], and dataset distillation synthesizes tiny training sets that reproduce full\-data performance\[[11](https://arxiv.org/html/2607.26628#bib.bib11)\]\. Our aim is narrower and diagnostic: rather than propose a new selection or synthesis method, we isolate*which property of a context*drives TabPFN’s accuracy, using controlled construction and mixed\-effects inference\[[14](https://arxiv.org/html/2607.26628#bib.bib14)\]\.

Benchmark data\.All experiments use public tasks from OpenML\[[15](https://arxiv.org/html/2607.26628#bib.bib15)\], which supplies the standardized, citable datasets that make the study reproducible\.

## 3Setup

We use TabPFN v3 withignore\_pretraining\_limits=True\. Each dataset is split 80/20 \(stratified, fixed split seed\)\. A context of sizekkis formed by selectingkktraining rows and fitting TabPFN on exactly those rows \(clf\.fit\(X\[idx\], y\[idx\]\)\); the held\-out 20% is the test set throughout\. We report ROC AUC \(one\-vs\-rest macro for the multiclass dataset\)\. For the stability and correlational analyses we draw20 independent random contextsperkk\. For the causal analyses in H2 we additionally*construct*contexts with targeted geometry \(Section on the controlled test\) rather than sampling them; for selection\-method and stress tests we use three or more seeds\.

Table 1:Datasets \(all small; public OpenML\[[15](https://arxiv.org/html/2607.26628#bib.bib15)\]\)\.Context descriptors\.Let the full training set have standardized feature matrixX∈ℝn×mX\\in\\mathbb\{R\}^\{n\\times m\}with column meansX¯\\bar\{X\}, and let a contextCCof sizekkhave feature rows\{xi\}i∈C\\\{x\_\{i\}\\\}\_\{i\\in C\}with meanX¯C\\bar\{X\}\_\{C\}\. We characterizeCCwith two families of measures\.

*Representativeness*\(how close to the full distribution\): the total\-variation distance between the context’s class distributionpCp\_\{C\}and the full class distributionpp, and the mean absolute per\-feature mean shift,

class\_tv=12​∑c\|pc−pC,c\|,feat\_shift=1m​∑j=1m\|X¯C,j−X¯j\|\.\\texttt\{class\\\_tv\}=\\tfrac\{1\}\{2\}\\sum\_\{c\}\\lvert p\_\{c\}\-p\_\{C,c\}\\rvert,\\qquad\\texttt\{feat\\\_shift\}=\\frac\{1\}\{m\}\\sum\_\{j=1\}^\{m\}\\bigl\\lvert\\bar\{X\}\_\{C,j\}\-\\bar\{X\}\_\{j\}\\bigr\\rvert\.\(1\)Smaller values meanCClooks more like the whole training set\.

*Diversity / coverage*\(how much of the spaceCCspans\): the mean pairwise Euclidean distance among context rows and the log\-determinant of the context covarianceΣC\\Sigma\_\{C\}\(a volume\),

diversity=1\|C\|2​∑i,i′∈C∥xi−xi′∥2,logdet\_cov=log​det\(ΣC\+ε​I\)\.\\texttt\{diversity\}=\\frac\{1\}\{\\lvert C\\rvert^\{2\}\}\\sum\_\{i,i^\{\\prime\}\\in C\}\\lVert x\_\{i\}\-x\_\{i^\{\\prime\}\}\\rVert\_\{2\},\\qquad\\texttt\{logdet\\\_cov\}=\\log\\det\\\!\\bigl\(\\Sigma\_\{C\}\+\\varepsilon I\\bigr\)\.\(2\)These are standard geometric coverage measures\. Mean pairwise distance summarizes spread, and the covariance log\-determinant is the standard volume of the region a point set occupies \(it appears, for example, in determinantal point processes\[[8](https://arxiv.org/html/2607.26628#bib.bib8)\]and coreset constructions\[[9](https://arxiv.org/html/2607.26628#bib.bib9),[10](https://arxiv.org/html/2607.26628#bib.bib10)\]\)\. The two families can vary independently, and separating them is the crux of H2\.

## 4H1: Context Stability

For eachkkwe drew 20 independent random contexts and recorded the resulting test AUC\. Table[2](https://arxiv.org/html/2607.26628#S4.T2)reports the mean±\\pm95% CI, and Table[3](https://arxiv.org/html/2607.26628#S4.T3)the coefficient of variationCV=σ/μ\\mathrm\{CV\}=\\sigma/\\mu, a scale\-free measure of variability, as a percentage\. Figure[1](https://arxiv.org/html/2607.26628#S4.F1)plots mean AUC and CV againstkk\.

Table 2:Mean ROC AUC±\\pm95% CI across 20 random contexts perkk\. Dashes:k=256k\{=\}256exceeds the training\-set size on the smallest datasets\.Table 3:Coefficient of variation \(CV=σ/μ\\mathrm\{CV\}=\\sigma/\\mu, %\) of ROC AUC across the same 20 random contexts\. CV falls askkgrows on every dataset\.![Refer to caption](https://arxiv.org/html/2607.26628v1/x1.png)Figure 1:Mean AUC \(left\) rises with context sizekk; the coefficient of variation across 20 random draws \(right\) falls withkkon every dataset\.What we observe\.On every one of the 15 datasets, variability across random contexts falls askkgrows\. On datasets with accuracy headroom \(diabetes, credit\-g, sonar, kc1, waveform, and others\) the CV drops from roughly 6–18% atk=16k\{=\}16to about 1–4% at the largestkk; datasets already near ceiling \(breast\-w, segment\) start low and still shrink\. Mean AUC rises withkkin parallel\. So on small data the context size is not only an accuracy knob but a*reliability*knob: a small context makes the model’s answer depend heavily on which rows happened to be drawn, and enlarging the context removes most of that dependence\. The most variable configuration we saw was kc1 atk=16k\{=\}16\(CV17\.8%17\.8\\%\); atk=16k\{=\}16diabetes AUC over 20 draws ranged from 0\.503 to 0\.839\.

## 5H2: What makes a context good, representativeness or diversity?

H2 as we first posed it, that a context which preserves the full training distribution scores higher, turns out to be the wrong reading of a real correlation\. This section builds the case in three steps: the correlation \(observed\), a controlled test that overturns its causal interpretation, and a disentangling experiment that identifies context*diversity/coverage*, not distribution matching, as the factor that actually tracks accuracy\.

Using the 20 random draws perkkfrom H1, we first ask whether the draws that better preserve the full training distribution score higher\. For each draw we computed two distances to the full training set: the total\-variation distance between class distributions \(class\_tv\) and the mean absolute standardized feature\-mean shift \(feat\_shift\)\. Table[4](https://arxiv.org/html/2607.26628#S5.T4)reports Pearsonrr, Spearmanρ\\rho, and linearR2R^\{2\}per dataset, pooled overkk\.

Table 4:Correlation between distribution distance and AUC \(all draws pooled overkk, per dataset\)\. Negative values mean that a context further from the full distribution scores lower, as hypothesized\.Table 5:Mean AUC of the most\- vs\. least\-representative third of random draws, split byclass\_tv\. The most\-representative third wins on all 15 datasets\.Datasetmost repr\.least repr\.Δ\\Deltavehicle0\.9530\.825\+0\.127\+0\.127sonar0\.8580\.746\+0\.112\+0\.112waveform\-50000\.9560\.867\+0\.088\+0\.088glass0\.9420\.864\+0\.077\+0\.077credit\-g0\.7520\.678\+0\.074\+0\.074diabetes0\.8510\.791\+0\.060\+0\.060…\(remaining 9 datasetsΔ\\Deltafrom\+0\.027\+0\.027down to\+0\.004\+0\.004\)breast\-w0\.9940\.990\+0\.004\+0\.004![Refer to caption](https://arxiv.org/html/2607.26628v1/x2.png)Figure 2:The correlational step \(later overturned as causal\)\. Left: Pearson correlation between feature\-mean shift and AUC across random draws per dataset \(all negative\)\. Right: mean AUC of the most\- vs\. least\-representative third of random draws\. This association is real but confounded with diversity, as the controlled experiment shows\.The correlation\.Across random draws, the direction predicted by H2 holds everywhere: all 15 correlations in Table[4](https://arxiv.org/html/2607.26628#S5.T4)are negative \(further from the full distribution→\\rightarrowlower AUC\), and in Table[5](https://arxiv.org/html/2607.26628#S5.T5)the most\-representative third of draws beats the least\-representative third on all 15 datasets\.feat\_shiftis the stronger predictor \(pooled Pearson−0\.55\-0\.55,R2=0\.31R^\{2\}=0\.31\), and it is strong on several datasets, including waveform\-5000 \(r=−0\.83r=\-0\.83,R2=0\.68R^\{2\}=0\.68\), credit\-g \(r=−0\.80r=\-0\.80\), vehicle \(r=−0\.79r=\-0\.79\), and spambase \(r=−0\.78r=\-0\.78\), while weaker near ceiling \(breast\-w\) or on noisy datasets \(kc1, heart\-statlog\)\. Taken alone, this looks like support for H2\. It is not, and the rest of this section explains why\.

### 5\.1A controlled test overturns the causal reading

Correlation among random draws cannot separate cause from confound\. We therefore*constructed*contexts at target representativeness levels rather than waiting for random draws to vary: a greedy procedure that adds rows to keep the running feature\-mean as close as possible to the full\-data mean \(“high” representativeness, very lowfeat\_shift\), a uniform “random” baseline, and a farthest\-from\-centroid procedure \(“low” representativeness, highfeat\_shift\), all with the class ratio matched\. These constructed contexts are designed to isolate causal factors, not to model practical sampling strategies; their purpose is to move one geometric property at a time\. If representativeness caused accuracy, the low\-shift contexts should score highest\.

The opposite happens \(Table[6](https://arxiv.org/html/2607.26628#S5.T6)\)\. Forcing lowfeat\_shiftreduces accuracy substantially, to 0\.49 on diabetes and 0\.39 on blood\-transfusion, far below random, while the high\-shift contexts are as good as or better than random\. On most datasets the high\-vs\-low effect is large and runs contrary to the original hypothesis \(Cohen’sddup to−22\.6\-22\.6, Cliff’sδ=−1\.0\\delta=\-1\.0on six datasets\)\. So matching the marginal feature means does not cause good performance; when isolated, it usually degrades it\.

Table 6:Controlled representativeness atk=128k\{=\}128\(15 contexts each, mean ROC AUC±\\pm95% CI\)\. Forcing low feature\-mean shift \(“high repr\.”\) reduces accuracy substantially;ddandδ\\deltaare the high\-vs\-low effect sizes\.The pattern is dominant but not universal: on 9 of 15 datasets forcing low shift reduces accuracy \(large negativedd,δ=−1\.0\\delta=\-1\.0on 6\), with the most dramatic drops on spambase \(0\.680\.68vs\.0\.980\.98\), waveform \(0\.700\.70vs\.0\.960\.96\), and the originals\. A handful of near\-ceiling datasets \(sonar, glass, wdbc\) show small positive effects\. The reason for the dominant pattern is a confound: our low\-shift construction also reduces the*diversity*of the context \(rows are chosen to cancel each other’s deviations, so they cluster\)\. The correlation in Table[4](https://arxiv.org/html/2607.26628#S5.T4)arose because, among random draws, a more shifted draw also tends to be a less diverse one\. To find which factor actually matters, we must vary them independently\.

### 5\.2Disentangling shift from diversity

We built six construction methods spanning the \(feat\_shift, diversity\) plane, ranging from a low\-shift/low\-diversity clump, to a low\-shift/*high*\-diversity set \(extremes paired to cancel in the mean\), to high\-shift/high\-diversity farthest\-point sampling\. For each context we recordedfeat\_shift, diversity \(mean pairwise distance\), and coverage \(log\-determinant of the context covariance\) alongside AUC\. We then computed*partial*correlations: the association of each factor with AUC while holding the other fixed \(Table[7](https://arxiv.org/html/2607.26628#S5.T7)\)\.

Table 7:Correlation of context geometry with AUC \(all six construction methods pooled, per dataset\)\. “shift∣\\middiv” is the partial correlation of feature\-mean shift with AUC controlling for diversity, and vice versa;β\\betaare standardized regression coefficients from AUC∼\\simshift\+\+diversity\.What we observe\.Diversity is the factor that survives in aggregate\. Pooled across datasets, diversity’s partial correlation with AUC \(controlling for shift\) is\+0\.22\+0\.22and its standardized coefficient\+0\.23\+0\.23, whereas feature\-mean shift’s partial correlation \(controlling for diversity\) is−0\.01\-0\.01and its coefficient−0\.01\-0\.01, essentially zero\. The per\-dataset picture is mixed rather than unanimous: diversity’s partial correlation is positive on nine of fifteen datasets \(strong on ionosphere\+0\.83\+0\.83, diabetes\+0\.60\+0\.60, wdbc\+0\.56\+0\.56\) and negative on a few \(australian, glass, heart\-statlog\), so the effect is directionally dominant, not universal\. Coverage \(log\-det of the context covariance\) tracks AUC about as strongly as pairwise diversity on the datasets where either does \(e\.g\.r=0\.72r=0\.72on vehicle,0\.690\.69on diabetes,0\.670\.67on segment\)\. The controlled construction \(Table[6](https://arxiv.org/html/2607.26628#S5.T6)\) remains the cleaner evidence: the low\-shift/high\-diversity set scores well while the low\-shift/low\-diversity clump scores poorly\.

We therefore correct H2\. The representativeness–accuracy correlation is real but*not causal*: it is a shadow of a diversity–accuracy relationship, because shifted random draws tend also to be narrow ones\. What a small\-data context needs appears to be*coverage of the feature space*, not a close match to the marginal feature distribution\. This also explains the cross\-method observation that K\-Means and FPS, which are far from distribution\-preserving but cover the space well, match or beat random \(H3\): they succeed through diversity, the factor that matters, not through representativeness, the factor that does not\.Formal inference\.To test this with proper accounting for dataset\-level variation, we fit a linear mixed\-effects model\[[14](https://arxiv.org/html/2607.26628#bib.bib14)\]over all 945 constructed contexts from the 15 datasets,

AUC∼diversity\+shift\+\(1∣dataset\),\\text\{AUC\}\\sim\\text\{diversity\}\+\\text\{shift\}\+\(1\\mid\\text\{dataset\}\),with predictors and outcome z\-scored within dataset\. Diversity is a highly significant predictor \(β=\+0\.23\\beta=\+0\.23, SE0\.030\.03,p=3×10−12p=3\\times 10^\{\-12\}, 95% CI\[\+0\.16,\+0\.29\]\[\+0\.16,\+0\.29\]\), while feature\-mean shift is not significant once diversity is in the model \(β=−0\.01\\beta=\-0\.01, SE0\.030\.03,p=0\.71p=0\.71, 95% CI\[−0\.08,\+0\.05\]\[\-0\.08,\+0\.05\]\)\. A random\-slope variant \(diversity slope varying by dataset\) agrees on the sign and significance of diversity \(β=\+0\.25\\beta=\+0\.25,p=0\.004p=0\.004\); shift is small and, if anything, negative \(β=−0\.09\\beta=\-0\.09,p=0\.008p=0\.008\), the opposite of what representativeness would predict\. With 15 dataset groups the random\-slope covariance is well\-conditioned, unlike the six\-dataset pilot\. The pooled effect is smaller than that pilot suggested \(β=0\.23\\beta=0\.23vs\.0\.470\.47\), which we read as regression to a more reliable estimate on the broader sample\.

We say “appears to be” deliberately: the per\-dataset partial correlations are mixed \(Table[7](https://arxiv.org/html/2607.26628#S5.T7)\), our diversity and coverage measures are geometric proxies, and the evidence is a controlled association across constructions, not a randomized single\-factor intervention\. The claim we stand behind is the aggregate one, that diversity predicts accuracy and feature\-mean shift does not, and it is backed by the controlled test and the mixed\-effects model\.

![Refer to caption](https://arxiv.org/html/2607.26628v1/x3.png)Figure 3:H2 corrected\. Left: across all constructed contexts, AUC tends to rise with context diversity \(z\-scored within dataset\)\. Right: partial correlations per dataset\. Diversity controlling for shift \(green\) is positive on most datasets, while shift controlling for diversity \(red\) is near zero or negative\. In aggregate \(pooled and mixed\-effects\) diversity is the factor that survives\.

## 6H3: Selection Cost

At a fixed budget ofk=128k\{=\}128we compared uniform random selection against K\-Means \(nearest training point to each centroid, using k\-means\+\+ seeding\[[6](https://arxiv.org/html/2607.26628#bib.bib6)\]\) and farthest\-point sampling \(FPS, the classic 2\-approximation tokk\-center\[[7](https://arxiv.org/html/2607.26628#bib.bib7)\]\), three seeds each\. Table[8](https://arxiv.org/html/2607.26628#S6.T8)reports mean AUC and the mean time to*choose*the prototypes \(separate from inference\)\.

Table 8:Selection methods atk=128k\{=\}128\(mean ROC AUC±\\pm95% CI over three seeds\)\.![Refer to caption](https://arxiv.org/html/2607.26628v1/x4.png)Figure 4:Selection methods atk=128k\{=\}128\. Accuracy is nearly tied across methods \(left\); selection cost differs by two to three orders of magnitude \(right\)\.What we observe\.Across all 15 datasets, accuracy differences between the three methods are small, within about one to three AUC points, and not in a consistent direction \(K\-Means marginally ahead on some, FPS or random on others\)\. The 95% confidence intervals overlap across methods on most datasets; on credit\-g, where K\-Means led random by about one point, the intervals \(0\.755±\.0680\.755\{\\pm\}\.068vs\.0\.745±\.0570\.745\{\\pm\}\.057\) overlap heavily, so that gap is not distinguishable from seed noise\. Meanwhile the compute cost differs sharply: random selection took on the order of0\.00030\.0003s versus∼\\sim0\.22 s for K\-Means, roughly three orders of magnitude\. Given near\-equal accuracy, the extra machinery of K\-Means or FPS bought little over drawing rows at random\.

This ties back to H2\. K\-Means and FPS already maximize context diversity, by spreading prototypes across clusters or across the feature space, which is exactly the factor H2 identifies as driving accuracy\. They perform similarly to random not despite ignoring distribution matching, but because random already supplies comparable coverage in expectation\. The methods differ mainly in cost, not in the property that matters\.

## 7Supporting: Redundancy and Robustness

Starting from a random context ofk=128k\{=\}128, we either duplicated a fraction of its rows \(adding no new information\) or dropped a fraction of them\.

Redundancy\.Duplicating rows had little effect on most of the 15 datasets: on 13 of them AUC stayed within about 1\.5 points up to 70% duplication\. The clear exception was blood\-transfusion, where duplication steadily hurt \(0\.686→0\.5830\.686\\rightarrow 0\.583\), with kc1 and glass showing milder declines \(≈−0\.015\\approx\-0\.015\)\. So duplicate context rows are mostly harmless when the signal is strong, but on weak\-signal datasets they skew the context and degrade predictions\.

Robustness\.Dropping rows degraded accuracy gradually rather than sharply\. Removing 50% of a 128\-row context cost under 1 AUC point on the stronger datasets \(breast\-w, diabetes, segment, wdbc\) and more on the small or weak ones \(sonar−0\.076\-0\.076, glass−0\.045\-0\.045, blood\-transfusion−0\.071\-0\.071\)\. This is consistent with H1: with fewer rows the context covers less of the space and predictions get both worse and noisier\.

## 8Discussion

The results fit together into one account of context sampling on small data\. H1 shows that a small context makes TabPFN’s prediction unstable, since the answer depends on which rows were drawn, and that enlarging the context removes most of that instability while also raising accuracy\. H2 asks why some contexts are better, and the controlled experiments give a sharper answer than the initial correlation suggested: what matters is the context’s*diversity / coverage*of the feature space, not how closely it matches the training distribution\. Matching the marginals, when forced in isolation, actually hurts \(because it reduces coverage\); diversity is the factor that survives once the two are separated\. H3 then follows naturally: uniform random sampling covers the space well in expectation, so it already captures the factor that matters, and the more expensive selection methods \(also diverse, but no more so in ways that help\) add little accuracy at much higher cost\.

A largerkkhelps in the same terms: more rows cover more of the space and are less subject to an unlucky, narrow draw\. Taken together, on small tabular datasets the useful things to control are the*size and diversity*of the context, not its distributional fidelity and not the sophistication of the selection algorithm\. Uniform random sampling at a sufficiently largekkis a strong and cheap default because it delivers both\.

## 9Conclusion

We investigated what makes an in\-context training set good for TabPFN on small tabular datasets\. \(1\) Larger contexts are both more accurate and substantially more stable, with the coefficient of variation of AUC falling several\-fold askkgrows\. \(2\) Accuracy correlates with how representative a random context is, but a controlled experiment overturns that reading: constructing contexts to match the feature means*lowers*accuracy \(by up to 0\.5 AUC\), and once we disentangle the two factors, context diversity/coverage predicts accuracy \(mixed\-effectsβ=\+0\.23\\beta=\+0\.23,p=3×10−12p=3\\times 10^\{\-12\}\) while feature\-mean shift does not \(β=−0\.01\\beta=\-0\.01,p=0\.71p=0\.71\)\. \(3\) K\-Means and farthest\-point selection, diverse but not distribution\-preserving, match random within about one AUC point at two to three orders of magnitude more cost\. The practical recommendation is direct: use a sufficiently large random context and spend no effort on expensive selection\. The scientific takeaway is that what a small\-data context needs appears to be coverage of the feature space, not fidelity to its distribution\. Random sampling succeeds because it provides that coverage for free\.

## 10Limitations

Several limitations bound how far these conclusions should be read\. The most important concerns the strength of the causal claim\. Our evidence that diversity, not representativeness, drives accuracy comes from controlled constructions, partial correlations, and a mixed\-effects model\. This is stronger than a raw correlation, but it is still an association over a designed set of contexts rather than a randomized intervention on a single factor\. The per\-dataset partial correlations are mixed \(positive on nine of fifteen datasets\), so the diversity effect is directionally dominant in aggregate but not universal; we say diversity “appears to be” the driver accordingly\. Diversity and coverage are themselves measured with geometric proxies \(mean pairwise distance, covariance log\-determinant\), and a different proxy could shift the details\.

The scope of the evidence is also narrow in ways worth stating plainly\. We study fifteen small tabular datasets \(roughly 200–5000 rows\), so the findings may not extend to much larger, higher\-dimensional, or non\-tabular data, and the strength of H2 on any single dataset depends on its having AUC headroom\. We use a single model \(TabPFN v3\); whether the same context behavior holds for other in\-context tabular learners is untested\. We compare three selection methods \(random, K\-Means, FPS\), so a method specifically optimized for context quality, for instance an active\-learning\-style informativeness criterion\[[13](https://arxiv.org/html/2607.26628#bib.bib13)\], could behave differently\. Our representativeness descriptors are two coarse distances \(class\-TV, feature\-mean shift\), and richer divergences \(e\.g\. MMD, per\-feature KL\) might explain more; we leave those to future work\. Finally, all runs used a single consumer GPU \(RTX 4060\), so the selection\-time comparison reflects that setup and would differ on other hardware\.

## Reproducibility

All experiments are fully reproducible\. The complete source code, experiment scripts, analysis scripts, and figure\-generation pipeline are available at:

The raw outputs of every experiment are stored underexperiments/results/, includingexp14\_context\_sampling\_\*\.json,exp15\_controlled\_\*\.json, andexp16\_disentangle\_\*\.json\. All tables and figures in this paper are generated directly from these result files using the provided analysis scripts \(analyze\_exp14\.py,analyze\_exp15\.py,analyze\_exp16\.py,mixed\_effects\_exp16\.py, andgenerate\_paper2\_figures\.py\), ensuring that every reported number is derived automatically from the logged experiment outputs\.

All datasets are publicly available through OpenML\. Dataset identifiers, random seeds, train/test splits, and experiment configurations are recorded in the result files\. Experiments were performed using TabPFN v3 on an NVIDIA RTX 4060 GPU\.

## References

- \[1\]Hollmann, N\., Müller, S\., Purucker, L\., Krishnakumar, A\., Körfer, M\., Hoo, S\. B\., Schirrmeister, R\. T\., & Hutter, F\. \(2025\)\. Accurate predictions on small data with a tabular foundation model\.Nature, 637\(8045\), 319–326\. DOI: 10\.1038/s41586\-024\-08328\-6\.
- \[2\]Müller, S\., Hollmann, N\., Pineda Arango, S\., Grabocka, J\., & Hutter, F\. \(2022\)\. Transformers can do Bayesian inference\. InInternational Conference on Learning Representations \(ICLR\)\.
- \[3\]Brown, T\. B\., Mann, B\., Ryder, N\., Subbiah, M\., et al\. \(2020\)\. Language models are few\-shot learners\.Advances in Neural Information Processing Systems \(NeurIPS\), 33, 1877–1901\.
- \[4\]Dong, Q\., Li, L\., Dai, D\., Zheng, C\., et al\. \(2024\)\. A survey on in\-context learning\. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)\.
- \[5\]García, S\., Derrac, J\., Cano, J\. R\., & Herrera, F\. \(2012\)\. Prototype selection for nearest neighbor classification: taxonomy and empirical study\.IEEE Transactions on Pattern Analysis and Machine Intelligence, 34\(3\), 417–435\.
- \[6\]Arthur, D\., & Vassilvitskii, S\. \(2007\)\. k\-means\+\+: the advantages of careful seeding\. InProceedings of the 18th Annual ACM–SIAM Symposium on Discrete Algorithms \(SODA\), 1027–1035\.
- \[7\]Gonzalez, T\. F\. \(1985\)\. Clustering to minimize the maximum intercluster distance\.Theoretical Computer Science, 38, 293–306\.
- \[8\]Kulesza, A\., & Taskar, B\. \(2012\)\. Determinantal point processes for machine learning\.Foundations and Trends in Machine Learning, 5\(2–3\), 123–286\.
- \[9\]Feldman, D\. \(2020\)\. Introduction to core\-sets: an updated survey\.arXiv preprint arXiv:2011\.09384\.
- \[10\]Sener, O\., & Savarese, S\. \(2018\)\. Active learning for convolutional neural networks: a core\-set approach\. InInternational Conference on Learning Representations \(ICLR\)\.
- \[11\]Wang, T\., Zhu, J\.\-Y\., Torralba, A\., & Efros, A\. A\. \(2018\)\. Dataset distillation\.arXiv preprint arXiv:1811\.10959\.
- \[12\]Ghorbani, A\., & Zou, J\. \(2019\)\. Data Shapley: equitable valuation of data for machine learning\. InProceedings of the 36th International Conference on Machine Learning \(ICML\), 2242–2251\.
- \[13\]Settles, B\. \(2009\)\. Active learning literature survey\. Computer Sciences Technical Report 1648, University of Wisconsin–Madison\.
- \[14\]Bates, D\., Mächler, M\., Bolker, B\., & Walker, S\. \(2015\)\. Fitting linear mixed\-effects models using lme4\.Journal of Statistical Software, 67\(1\), 1–48\.
- \[15\]Vanschoren, J\., van Rijn, J\. N\., Bischl, B\., & Torgo, L\. \(2013\)\. OpenML: networked science in machine learning\.ACM SIGKDD Explorations Newsletter, 15\(2\), 49–60\.