GIST: Generator-state-Informed Supervision for Tabular Foundation Models

# GIST: Generator-state-Informed Supervision for Tabular Foundation Models
Source: [https://arxiv.org/html/2610.12904](https://arxiv.org/html/2610.12904)
###### Abstract

Tabular foundation models \(TFMs\) based on prior\-data fitted networks typically use sampled hard query labels, leaving avoidable sampling variability in pretraining\. A key observation is that intermediate generator states can reveal label uncertainty beyond a sampled outcome\. We proposeGenerator\-state\-InformedSupervision forTFMs \(GIST\), which uses these states to average query losses over conditional label distributions\. GIST jointly designs task generation and query supervision, using entropy\-calibrated encodings to obtain tractable, input\-consistent targets on supported label\-dependent branches\. We show that this averaging preserves the population objective and yields no larger gradient covariance at fixed parameters within each generating distribution\. Under matched pretraining schedules, GIST improves mean rank across three architectures on both TabArena and TALENT without changing architecture or inference\. Across these benchmarks and architectures, GIST wins 62\.1% of paired dataset\-level comparisons against the corresponding hard\-label controls\.

## 1Introduction

Figure 1:Overview of GIST\.\(I\) Task generation retains state and calibrates supported category encodings\. \(II\) Query targets use conditional probabilities, with hard\-label fallback\. \(III\) The TFM uses standard inputs and hard context labels\. Objective and raw\-gradient covariance comparisons use the same generating distribution at fixed parameters\.Tabular prediction estimates a categorical or numerical target from a row of features, supporting applications such as credit assessment and clinical risk prediction\. Tabular foundation models \(TFMs\) such as TabPFN and TabICL predict targets for query rows through in\-context learning, using labeled rows as context\([Hollmann et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib2);[Qu et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib3)\)\. Many follow the prior\-data fitted network \(PFN\) approach: a generator samples a data\-generating mechanism from a prior, produces a synthetic table, and supplies context and query rows for pretraining\([Müller et al\., 2022](https://arxiv.org/html/2610.12904#bib.bib28)\)\.

Synthetic priors determine the feature\-label relationships encountered during pretraining\. Priors based on Bayesian neural networks \(BNNs\) and structural causal models \(SCMs\) provide synthetic tasks for TabPFN\([Hollmann et al\., 2023](https://arxiv.org/html/2610.12904#bib.bib1)\), while mixtures of neural and tree\-based SCMs supply tasks for TabICL\([Qu et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib3)\)\. Randomized graph\-based SCMs in TabICLv2\([Qu et al\., 2026](https://arxiv.org/html/2610.12904#bib.bib4)\)and mixtures of complementary priors in Mitra\([Zhang et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib5)\)further diversify the generated tasks\.

Standard PFN pretraining evaluates query predictions against sampled hard labels, yielding an unbiased Monte Carlo estimate of the population loss and its gradient\([Müller et al\., 2022](https://arxiv.org/html/2610.12904#bib.bib28);[Nagler, 2023](https://arxiv.org/html/2610.12904#bib.bib29)\)\. Although synthetic tasks can be generated continually, the model processes only a finite number within its computational budget\. For a fixed input, a sampled label selects one class’s loss even when the generator can provide conditional probabilities for all classes\. This raises our central research question:*Can we use information retained during synthetic task generation to reduce the variability introduced by query\-label sampling during pretraining?*

We proposeGenerator\-state\-InformedSupervision forTFMs \(GIST\), a framework thatcouples synthetic task generation with conditional query supervision\. Its statistical basis is Rao\-Blackwellization \(RB\)\([Casella and Robert, 1996](https://arxiv.org/html/2610.12904#bib.bib33)\): replacing the sampled hard\-label query loss with its conditional expectation given the model input and selected generator information\. This conditioning fixes the prediction, allowing GIST to average over the remaining query\-label uncertainty whenever the conditional probabilities are tractable\. Under a fixed generating distribution, we show that this averaging preserves the population objective and yields no larger gradient covariance at fixed model parameters\. The retained generator information is used only to construct targets; the predictor receives standard context and query inputs \(Figure[1](https://arxiv.org/html/2610.12904#S1.F1)\)\.

The target construction depends on how the sampled label enters input generation\. For a query\-label draw that does not affect the model input or task acceptance, the sampling probabilities give the target directly\. If the sampled class influences generated features, those features provide additional evidence about the label\. For supported label\-dependent branches, GIST represents the sampled class by a category code perturbed with Gaussian noise and routes all label\-dependent feature computations through this encoding\. Its local likelihood then yields the posterior target without evaluating a marginal likelihood through the downstream graph\. An entropy budget calibrates the noise scale by bounding average residual label uncertainty in this channel\. This encoding changes the affected branch’s feature\-label distribution; unsupported branches retain hard\-label supervision\.

To assess*whether GIST improves predictive performance across architectures*, we compare it with original\-prior hard\-label controls under matched pretraining schedules on TabICLv2\([Qu et al\., 2026](https://arxiv.org/html/2610.12904#bib.bib4)\), TabPFN\-2\.5\([Grinsztajn et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib23)\), and TabSwift\([Liu and Ye, 2026](https://arxiv.org/html/2610.12904#bib.bib8)\)\. Evaluation covers IID classification on TabArena and TALENT\([Erickson et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib21);[Liu et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib25)\), and temporal and grouped OOD tasks on BeyondArena\([Purucker et al\., 2026](https://arxiv.org/html/2610.12904#bib.bib27)\)\. At the final matched checkpoints, GIST improves mean rank and wins on a majority of datasets in all six IID architecture–benchmark combinations\. We separately examine*where and when conditional supervision reduces clipped\-gradient variance*, using fixed\-checkpoint diagnostics under a shared calibrated prior across training stages, table lengths, and model components\. Finally, a TabSwift budget sweep assesses*how sensitive predictive performance is to the entropy budget*\. Our contributions are as follows:

- •Wejointly design synthetic task generation and query supervision, combining direct conditional targets with entropy\-calibrated Gaussian encodings and their posterior targets\.
- •We establishpopulation\-objective preservation and a gradient covariance reduction identityfor conditional supervision with hard\-label fallback under a fixed generating distribution\.
- •Experiments on over 200 tabular datasets from TabArena and TALENT show that GIST improves predictive performance across three representative TFM architectures, winning62\.1%of paired dataset\-level comparisons against the corresponding hard\-label controls\.

## 2Preliminaries and Problem Setup

We study PFN\-style tabular foundation models pretrained on synthetic classification tasks\([Hollmann et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib2);[Qu et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib3)\)\. A generator samples a task mechanismT∼ΠT\\sim\\Piand a context sizeN∼ΠNN\\sim\\Pi\_\{N\}, then produces a labeled contextDN=\{\(Xi,Yi\)\}i=1ND\_\{N\}=\\\{\(X\_\{i\},Y\_\{i\}\)\\\}\_\{i=1\}^\{N\}and a query\(X,Y\)\(X,Y\)\. The task priorΠ\\Pispecifies a distribution over data\-generating mechanisms, such as structural causal models and their parameters\. The predictor receivesS=\(DN,X,N\)S=\(D\_\{N\},X,N\)and outputs class probabilitiesqθ\(⋅∣S\)q\_\{\\theta\}\(\\cdot\\mid S\)overKKclasses\. We use one query for notation; with multiple queries,SSincludes the full visible input and the loss is averaged over query labels\. Unless stated otherwise, expectations and conditional distributions are taken under the pretraining distribution after preprocessing and task filtering\.

The population objective is the expected hard\-label cross\-entropy\([Nagler, 2023](https://arxiv.org/html/2610.12904#bib.bib29)\)\. Formmindependent synthetic episodes, its unbiased Monte Carlo estimate is

R\(θ\)=𝔼\[−logqθ\(Y∣S\)\],R^m\(θ\)=−1m∑j=1mlogqθ\(Yj∣Sj\)\.R\(\\theta\)=\\mathbb\{E\}\[\-\\log q\_\{\\theta\}\(Y\\mid S\)\],\\qquad\\widehat\{R\}\_\{m\}\(\\theta\)=\-\\frac\{1\}\{m\}\\sum\_\{j=1\}^\{m\}\\log q\_\{\\theta\}\(Y\_\{j\}\\mid S\_\{j\}\)\.\(1\)Heremmcounts synthetic episodes, whereasNNcounts context observations\. Task, input, and query\-label sampling contribute to gradient variance\.

LetZZcollect the full model inputSSand selected information retained during task generation, such as task parameters and intermediate encodings\. Define

pZ​\(k\)=P⁡\(Y=k∣Z\),πS​\(k\)=P⁡\(Y=k∣S\)=𝔼⁡\[pZ​\(k\)∣S\]\.p\_\{Z\}\(k\)=P\(Y=k\\mid Z\),\\qquad\\pi\_\{S\}\(k\)=P\(Y=k\\mid S\)=\\mathbb\{E\}\[p\_\{Z\}\(k\)\\mid S\]\.\(2\)The posterior predictive distributionπS\\pi\_\{S\}minimizesRRamong all predictors that depend only onSS\([Nagler, 2023](https://arxiv.org/html/2610.12904#bib.bib29)\)\. The conditional query distributionpZp\_\{Z\}supplies supervision when tractable, while the predictor receives onlySS\. At a fixed inputSS, averaging these targets over the additional generator information recoversπS\\pi\_\{S\}\. Our goal is to construct synthetic tasks and retained states that make these conditional targets tractable on supported branches \(Section[3](https://arxiv.org/html/2610.12904#S3)\)\. Within each fixed generating distribution, we compare conditional averaging with hard\-label supervision for the same population objectiveR⁡\(θ\)R\(\\theta\)\.

## 3Method

GIST constructs conditional query supervision from retained generator information\. We first define the supervision rule and identify when its targets are directly available, then construct and calibrate an encoding for supported label\-dependent branches\. We conclude with the combined training objective and its statistical guarantees\.

### 3\.1Generator\-informed query supervision

LetZZretain the complete model inputSSand selected generator information, and writepZ​\(k\)=P⁡\(Y=k∣Z\)p\_\{Z\}\(k\)=P\(Y=k\\mid Z\)\(Section[2](https://arxiv.org/html/2610.12904#S2)\)\. For a query with hard\-label lossℓH​\(θ,S,Y\)=−log⁡qθ​\(Y∣S\)\\ell\_\{\\mathrm\{H\}\}\(\\theta;S,Y\)=\-\\log q\_\{\\theta\}\(Y\\mid S\), GIST uses the conditional loss wheneverpZp\_\{Z\}is tractable:

ℓRB\(θ;Z\):=𝔼\[ℓH\(θ;S,Y\)∣Z\]=−∑k=1KpZ\(k\)logqθ\(k∣S\)\.\\ell\_\{\\mathrm\{RB\}\}\(\\theta;Z\):=\\mathbb\{E\}\[\\ell\_\{\\mathrm\{H\}\}\(\\theta;S,Y\)\\mid Z\]=\-\\sum\_\{k=1\}^\{K\}p\_\{Z\}\(k\)\\log q\_\{\\theta\}\(k\\mid S\)\.\(3\)GivenZZ, the model prediction is fixed, soℓRB\\ell\_\{\\mathrm\{RB\}\}averages the hard\-label loss over the remaining query\-label uncertainty\. Proposition[1](https://arxiv.org/html/2610.12904#Thmproposition1)shows that this Rao\-Blackwellization preserves the population objective and yields no larger gradient covariance under a fixed generating distribution and at fixed model parameters\. The construction ofpZp\_\{Z\}must account for the final input, so it depends on whether the sampled query class enters input generation\.

During table synthesis, the generator samples a label for each query row\. We call this query\-label draw*terminal*when, given the retained pre\-draw state, its outcome does not affect the model input\. Its sampling probabilities then give the conditional target directly, provided task filtering is also independent of the remaining query\-label randomness under the same conditioning \(Appendix[B\.1](https://arxiv.org/html/2610.12904#A2.SS1)\)\. This includes converters that pass a continuous state downstream independently of the sampled class, even when the node has observed descendants\. For MLP and tree generators with terminal target noise, we integrate that noise through class boundaries fitted on the context\.

When the sampled class affects observed features \(Figure[2](https://arxiv.org/html/2610.12904#S3.F2)\(I\)\), those features supply additional label evidence, so the pre\-draw probabilities must be updated\. Direct evaluation of this posterior can require inference through the downstream graph\. For supported label\-dependent branches, we jointly design the category encoding and its supervision rule so that the update can be computed from a local likelihood\. Branches without a supported conditional target retain sampled hard\-label supervision\.

### 3\.2Gaussian encoding for label\-dependent features

Figure 2:Gaussian encoding and entropy\-budget calibration\.\(I\) Observing label\-dependent features updates class probabilities\. \(II\) Routing all query\-label effects through a local Gaussian channel makes posterior targets tractable\. \(III\) Calibration selects the largest noise scale certified by an entropy budget; nearby codes require a smaller scale in the illustrated example\.In the supported label\-dependent converters, a sampled classYrY\_\{r\}is mapped to a real\-valued category codebYrb\_\{Y\_\{r\}\}for downstream feature generation\. GIST adds Gaussian noise to this code and routes all sampled\-class effects on downstream features through the resulting encodingWrW\_\{r\}\(Figure[2](https://arxiv.org/html/2610.12904#S3.F2)\(II\)\)\. Retaining this encoding allows the conditional target to be computed from its local likelihood\.

We describe the encoding channel before task filtering\. Letℋ\\mathcal\{H\}be the state before target\-category draws, containing a codebookB=\(b1,…,bK\)B=\(b\_\{1\},\\ldots,b\_\{K\}\), withbk∈ℝdb\_\{k\}\\in\\mathbb\{R\}^\{d\}, and row probabilitiesprp\_\{r\}\. ForL=N\+QL=N\+Qrows, comprisingNNcontext rows andQQqueries, we sample

Yr∣ℋ∼Categorical\(pr\),Wr=bYr\+σεr,εr∼𝒩\(0,Id\)\.Y\_\{r\}\\mid\\mathcal\{H\}\\sim\\operatorname\{Categorical\}\(p\_\{r\}\),\\qquad W\_\{r\}=b\_\{Y\_\{r\}\}\+\\sigma\\varepsilon\_\{r\},\\qquad\\varepsilon\_\{r\}\\sim\\mathcal\{N\}\(0,I\_\{d\}\)\.\(4\)Conditional onℋ\\mathcal\{H\}, the row labels are independent, and the encoding noises are independent across rows and independent of all labels\. The scaleσ=σ⁡\(ℋ\)\>0\\sigma=\\sigma\(\\mathcal\{H\}\)\>0depends only on pre\-draw information and is shared across rows and classes\. The sampled classYrY\_\{r\}remains the dataset label, while downstream nodes receive the noisy codeWrW\_\{r\}\. Section[3\.3](https://arxiv.org/html/2610.12904#S3.SS3)specifies how the scale is chosen fromℋ\\mathcal\{H\}\.

RetainZσ=\(ℋ,W1:L,Y1:N,S\)Z\_\{\\sigma\}=\(\\mathcal\{H\},W\_\{1:L\},Y\_\{1:N\},S\), excluding query labels and the realized encoding noiseε1:L\\varepsilon\_\{1:L\}\. Conditional onℋ\\mathcal\{H\}and the context labels, all query\-label effects on the input and task filtering pass throughW1:LW\_\{1:L\}\. Formally, under the channel law before task filtering,

YN\+1:L⟂\(S,E\)∣\(ℋ,W1:L,Y1:N\),Y\_\{N\+1:L\}\\perp\(S,E\)\\mid\(\\mathcal\{H\},W\_\{1:L\},Y\_\{1:N\}\),whereEEdenotes task acceptance\. Once these encodings are fixed, the downstream likelihood is common to all candidate query\-label vectors and cancels in Bayes’ rule\. The conditional target for queryr\>Nr\>Nis therefore

p~σ,r​\(k\)=pr​kexp\[−∥Wr−bk∥22/\(2σ2\)\]∑j=1Kpr​jexp\[−∥Wr−bj∥22/\(2σ2\)\]\.\\widetilde\{p\}\_\{\\sigma,r\}\(k\)=\\frac\{p\_\{rk\}\\exp\\\!\\left\[\-\\\|W\_\{r\}\-b\_\{k\}\\\|\_\{2\}^\{2\}/\(2\\sigma^\{2\}\)\\right\]\}\{\\sum\_\{j=1\}^\{K\}p\_\{rj\}\\exp\\\!\\left\[\-\\\|W\_\{r\}\-b\_\{j\}\\\|\_\{2\}^\{2\}/\(2\\sigma^\{2\}\)\\right\]\}\.\(5\)On accepted tasks,p~σ,r​\(k\)=P⁡\(Yr=k∣Zσ\)=pZσ,r​\(k\)\\widetilde\{p\}\_\{\\sigma,r\}\(k\)=P\(Y\_\{r\}=k\\mid Z\_\{\\sigma\}\)=p\_\{Z\_\{\\sigma\},r\}\(k\), supplying the conditional probabilities in Equation \([3](https://arxiv.org/html/2610.12904#S3.E3)\)\. The same channel generates the category representation and defines its supervision target\. This local encoding change modifies the branch’s feature\-label distribution\. Appendix[B\.2](https://arxiv.org/html/2610.12904#A2.SS2)proves the posterior identity after filtering, and Appendix[B\.4](https://arxiv.org/html/2610.12904#A2.SS4)gives the GraphSCM instantiations\.

### 3\.3Entropy\-budget calibration

The noise scale controls how much label uncertainty remains after the encoding is observed\. With distinct class codes, an almost deterministic encoding leaves little uncertainty to average, whereas excessive noise weakens the label information carried by the encoding\. A fixed scale may leave little uncertainty for well\-separated codes but obscure label information for nearby codes \(Figure[2](https://arxiv.org/html/2610.12904#S3.F2)\(III\)\)\. We therefore select the largest scale certified by an entropy budget, adapting it to the task’s class probabilities and codebook\.

Fixℋ=h\\mathcal\{H\}=hand, under the channel law before task filtering, define the local conditional entropyℰr​\(σ\)=H⁡\(Yr∣Wr,ℋ=h\)\\mathcal\{E\}\_\{r\}\(\\sigma\)=H\(Y\_\{r\}\\mid W\_\{r\},\\mathcal\{H\}=h\), averaging over labels and encoding noise\. For a candidate noise scale, we evaluate an entropy upper bound from the row probabilities and pairwise code distances before labels and encoding noise are sampled\. The Bhattacharyya lower bound on mixture entropy\([Kolchinsky and Tracey, 2017](https://arxiv.org/html/2610.12904#bib.bib30)\)yields

Ur​\(σ\)\\displaystyle U\_\{r\}\(\\sigma\):=H⁡\(pr\)\+∑ipr​i​log⁡\[∑jpr​j​exp⁡\(−‖bi−bj‖228​σ2\)\],\\displaystyle:=H\(p\_\{r\}\)\+\\sum\_\{i\}p\_\{ri\}\\log\\\!\\left\[\\sum\_\{j\}p\_\{rj\}\\exp\\\!\\left\(\-\\frac\{\\\|b\_\{i\}\-b\_\{j\}\\\|\_\{2\}^\{2\}\}\{8\\sigma^\{2\}\}\\right\)\\right\],\(6\)0\\displaystyle 0≤ℰr​\(σ\)≤Ur​\(σ\)≤H⁡\(pr\),\\displaystyle\\leq\\mathcal\{E\}\_\{r\}\(\\sigma\)\\leq U\_\{r\}\(\\sigma\)\\leq H\(p\_\{r\}\),whereH\(pr\)=−∑kpr​klogpr​kH\(p\_\{r\}\)=\-\\sum\_\{k\}p\_\{rk\}\\log p\_\{rk\}and logarithms are natural\.

Define the all\-row averagesU¯​\(s\)=L−1​∑r=1LUr​\(s\)\\overline\{U\}\(s\)=L^\{\-1\}\\sum\_\{r=1\}^\{L\}U\_\{r\}\(s\)andH¯=L−1​∑r=1LH⁡\(pr\)\\overline\{H\}=L^\{\-1\}\\sum\_\{r=1\}^\{L\}H\(p\_\{r\}\)\. Forβ∈\(0,1\]\\beta\\in\(0,1\], our experiments use the binary\-cap budgetℬβ​\(h\)=β​min⁡\{H¯,log⁡2\}\\mathcal\{B\}\_\{\\beta\}\(h\)=\\beta\\min\\\{\\overline\{H\},\\log 2\\\}\. The cap limits the residual\-entropy allowance toβ​log⁡2\\beta\\log 2whenH¯\>log⁡2\\overline\{H\}\>\\log 2\. Subject to an optional capσmax∈\(0,∞\]\\sigma\_\{\\max\}\\in\(0,\\infty\], we select the largest finite scale certified by this budget\. Uncapped calibration uses0<β<10<\\beta<1; whenβ=1\\beta=1, we require a finiteσmax\\sigma\_\{\\max\}\. When a positive finite maximizer exists,

σ⋆\(ℋ\)=max\{s≥0:s≤σmax,U¯\(s\)≤ℬβ\(ℋ\)\},\\sigma\_\{\\star\}\(\\mathcal\{H\}\)=\\max\\left\\\{s\\geq 0:\\ s\\leq\\sigma\_\{\\max\},\\quad\\overline\{U\}\(s\)\\leq\\mathcal\{B\}\_\{\\beta\}\(\\mathcal\{H\}\)\\right\\\},\(7\)using the right\-hand limit ats=0s=0\.

At a feasible positive scale, Equations \([6](https://arxiv.org/html/2610.12904#S3.E6)\) and \([7](https://arxiv.org/html/2610.12904#S3.E7)\) giveL−1​∑r=1Lℰr​\(σ⋆\)≤U¯​\(σ⋆\)≤ℬβ​\(h\)L^\{\-1\}\\sum\_\{r=1\}^\{L\}\\mathcal\{E\}\_\{r\}\(\\sigma\_\{\\star\}\)\\leq\\overline\{U\}\(\\sigma\_\{\\star\}\)\\leq\\mathcal\{B\}\_\{\\beta\}\(h\)\. Equivalently, the encoding retains at leastH¯−ℬβ​\(h\)≥\(1−β\)​H¯\\overline\{H\}\-\\mathcal\{B\}\_\{\\beta\}\(h\)\\geq\(1\-\\beta\)\\overline\{H\}of label information on average across all rows before task filtering\.

SinceU¯​\(s\)\\overline\{U\}\(s\)is continuous and nondecreasing fors\>0s\>0, calibration selects a feasible finite cap directly or locates the budget threshold by bisection\. Infeasible budgets retain the original encoding and sampled hard labels\. Other cases in which no positive scale is returned follow the branch rules in Appendix[C\.2](https://arxiv.org/html/2610.12904#A3.SS2)\. Appendices[C\.1](https://arxiv.org/html/2610.12904#A3.SS1)–[C\.4](https://arxiv.org/html/2610.12904#A3.SS4)give the derivations, numerical procedure, and alternative budget policy\.

### 3\.4Training objective and statistical guarantees

The preceding constructions supply a targettr=pZ,rt\_\{r\}=p\_\{Z,r\}on supported branches, using either terminal probabilities or the channel posterior, andtr=eYrt\_\{r\}=e\_\{Y\_\{r\}\}on hard\-label fallback branches\. HerepZ,r​\(k\)=P⁡\(Yr=k∣Z\)p\_\{Z,r\}\(k\)=P\(Y\_\{r\}=k\\mid Z\)uses a common retained stateZZfor the table, andeYre\_\{Y\_\{r\}\}is the sampled label’s one\-hot vector\. ForQQqueries with predictionsqθ,rq\_\{\\theta,r\}, GIST uses

ℓMtable\(θ\)=−1Q∑r=N\+1L∑k=1Ktr\(k\)logqθ,r\(k∣S\)\.\\ell\_\{\\mathrm\{M\}\}^\{\\mathrm\{table\}\}\(\\theta\)=\-\\frac\{1\}\{Q\}\\sum\_\{r=N\+1\}^\{L\}\\sum\_\{k=1\}^\{K\}t\_\{r\}\(k\)\\log q\_\{\\theta,r\}\(k\\mid S\)\.\(8\)Targets are held fixed during differentiation\. The loss reuses the model prediction; context labels remain hard, and the architecture and inference procedure are unchanged\.

For the single\-query losses in Section[3\.1](https://arxiv.org/html/2610.12904#S3.SS1), writeGa=∇θℓaG\_\{a\}=\\nabla\_\{\\theta\}\\ell\_\{a\}andVa=Cov⁡\(Ga\)V\_\{a\}=\\operatorname\{Cov\}\(G\_\{a\}\)\. LetPPdenote the joint law of\(S,Y,Z\)\(S,Y,Z\)induced by a specified generator, calibration rule, and task filter, independent ofθ\\theta\. The comparisons below usePPat fixed parameters and model randomness\. We assume integrable losses, finite gradient second moments, and interchangeability of differentiation and conditional expectation \(Appendix[A\.1](https://arxiv.org/html/2610.12904#A1.SS1)\)\.

###### Proposition 1\(Conditional supervision\)\.

IfpZ​\(k\)=P⁡\(Y=k∣Z\)p\_\{Z\}\(k\)=P\(Y=k\\mid Z\), then

𝔼⁡\[ℓRB​\(θ,Z\)\]=R⁡\(θ\),GRB=𝔼⁡\[GH∣Z\],VH−VRB=𝔼⁡\[Cov⁡\(GH∣Z\)\]⪰0\.\\mathbb\{E\}\[\\ell\_\{\\mathrm\{RB\}\}\(\\theta;Z\)\]=R\(\\theta\),\\quad G\_\{\\mathrm\{RB\}\}=\\mathbb\{E\}\[G\_\{\\mathrm\{H\}\}\\mid Z\],\\quad V\_\{\\mathrm\{H\}\}\-V\_\{\\mathrm\{RB\}\}=\\mathbb\{E\}\\\!\\left\[\\operatorname\{Cov\}\(G\_\{\\mathrm\{H\}\}\\mid Z\)\\right\]\\succeq 0\.\(9\)

###### Proof\.

GivenZZ, the model prediction is fixed, so averaging the hard\-label loss gives Equation \([3](https://arxiv.org/html/2610.12904#S3.E3)\)\. Taking expectations preservesRR, and differentiating gives the conditional\-gradient identity\. The covariance equality follows from the law of total covariance\. ∎

For selective supervision, letA=A⁡\(Z\)∈\{0,1\}A=A\(Z\)\\in\\\{0,1\\\}select tractable branches before the residual label draw, and writeΓθ​\(Z\)=Cov⁡\(GH∣Z\)\\Gamma\_\{\\theta\}\(Z\)=\\operatorname\{Cov\}\(G\_\{\\mathrm\{H\}\}\\mid Z\)\. The single\-query form of Equation \([8](https://arxiv.org/html/2610.12904#S3.E8)\) then satisfies

ℓM=A​ℓRB\+\(1−A\)​ℓH,𝔼⁡\[ℓM\]=R⁡\(θ\),VH−VM=𝔼⁡\[A​Γθ​\(Z\)\]⪰0\.\\ell\_\{\\mathrm\{M\}\}=A\\ell\_\{\\mathrm\{RB\}\}\+\(1\-A\)\\ell\_\{\\mathrm\{H\}\},\\quad\\mathbb\{E\}\[\\ell\_\{\\mathrm\{M\}\}\]=R\(\\theta\),\\quad V\_\{\\mathrm\{H\}\}\-V\_\{\\mathrm\{M\}\}=\\mathbb\{E\}\[A\\Gamma\_\{\\theta\}\(Z\)\]\\succeq 0\.\(10\)The mixed objective retains the posterior predictive targetπS\\pi\_\{S\}and covariance ordering underPP\(Appendix[B\.3](https://arxiv.org/html/2610.12904#A2.SS3)\)\. The table\-level extension uses a common retained state and includes cross\-query covariance terms \(Appendix[A\.5](https://arxiv.org/html/2610.12904#A1.SS5)\)\.

For softmax predictionsqθ=softmax⁡\(aθ\)q\_\{\\theta\}=\\operatorname\{softmax\}\(a\_\{\\theta\}\), letJθ=∂aθ/∂θ⊤J\_\{\\theta\}=\\partial a\_\{\\theta\}/\\partial\\theta^\{\\top\}be the logit Jacobian\. The removable covariance for a single query is

Γθ​\(Z\)=Jθ⊤​\[Diag⁡\(pZ\)−pZ​pZ⊤\]​Jθ\.\\Gamma\_\{\\theta\}\(Z\)=J\_\{\\theta\}^\{\\top\}\\left\[\\operatorname\{Diag\}\(p\_\{Z\}\)\-p\_\{Z\}p\_\{Z\}^\{\\top\}\\right\]J\_\{\\theta\}\.\(11\)The bracketed term is the conditional covariance of the one\-hot label, mapped to parameter space by the logit Jacobian\. Deterministic conditional labels giveΓθ​\(Z\)=0\\Gamma\_\{\\theta\}\(Z\)=0, while task and input variation remain after averaging\. Appendix[A](https://arxiv.org/html/2610.12904#A1)gives the covariance decomposition and gradient estimation error, along with an asymptotic comparison under additional M\-estimation assumptions\. Proposition[1](https://arxiv.org/html/2610.12904#Thmproposition1)concerns raw gradients; Section[4\.3](https://arxiv.org/html/2610.12904#S4.SS3)evaluates clipped\-gradient variability empirically\.

## 4Experiments

We evaluate GIST as a joint design of synthetic task generation and query supervision\. Our experiments address three complementary questions: \(i\) does the complete design improve downstream prediction across TFM architectures under matched pretraining schedules, in both IID and OOD evaluation? \(ii\) under a shared calibrated prior, where and when does conditional supervision reduce clipped\-gradient variability? \(iii\) how sensitive is downstream performance to the entropy budget controlling the encoding channel?

### 4\.1Experimental setup

Datasets\.IID evaluation covers 38 TabArena classification datasets and 188 TALENT datasets shared by all methods\([Erickson et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib21);[Liu et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib25)\)\. OOD evaluation uses BeyondArena’s temporal and grouped splits\([Purucker et al\., 2026](https://arxiv.org/html/2610.12904#bib.bib27)\), with each architecture’s complete feasible task set: 21 datasets for TabICLv2, 18 for TabSwift, and 12 for TabPFN\-2\.5\. IID and OOD refer to the real\-data train/test split regimes\.

Baselines\.We evaluate Hard and GIST on three representative TFM architectures: TabICLv2 with column\-then\-row encoding\([Qu et al\., 2026](https://arxiv.org/html/2610.12904#bib.bib4)\), TabSwift with efficient row\-wise attention\([Liu and Ye, 2026](https://arxiv.org/html/2610.12904#bib.bib8)\), and TabPFN\-2\.5 with cell\-based alternating row/column attention\([Grinsztajn et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib23)\)\. Within TabICLv2’s GraphSCM generator family, Hard uses the original generator and hard query labels; GIST uses our joint generator–supervision design\. We also include tuned CatBoost, LightGBM, XGBoost, RealMLP, and TabM baselines\([Prokhorenkova et al\., 2018](https://arxiv.org/html/2610.12904#bib.bib12);[Ke et al\., 2017](https://arxiv.org/html/2610.12904#bib.bib14);[Chen and Guestrin, 2016](https://arxiv.org/html/2610.12904#bib.bib13);[Holzmüller et al\., 2024](https://arxiv.org/html/2610.12904#bib.bib17);[Gorishniy et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib26)\)\.

Metrics\.We report ROC\-AUC for binary tasks and log loss for multiclass tasks, averaging within datasets before weighting datasets equally\. For cross\-task aggregation, error ise=1−AUCe=1\-\\mathrm\{AUC\}or log loss, respectively\. On each split, we compute Improvability\([Erickson et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib21)\)asIm=1−minj⁡ej/emI\_\{m\}=1\-\\min\_\{j\}e\_\{j\}/e\_\{m\}, withIm=0I\_\{m\}=0whenem=0e\_\{m\}=0; mean rank uses average ranks for ties\. Both are lower\-is\-better and follow the same dataset weighting\. Paired win rates compare dataset\-level mean scores within each architecture, retaining ties in the denominator\.

Training and evaluation protocols\.Within each architecture, Hard and GIST use matched effective global batch sizes, training schedules, and inference settings\. Stage 1 \(short\-context pretraining\) trains for 30,000 updates on synthetic tasks of fixed length 1,024 rows\. Stage 2 \(long\-context continuation\) trains for 6,000 additional updates from the corresponding Stage 1 checkpoint, with task lengths sampled logarithmically between 400 and 10,240 rows\. Task length counts both support and query rows\. Unless stated otherwise, Hard and GIST use the final checkpoints of this two\-stage, 36K\-update schedule\. By default, GIST usesβ=0\.15\\beta=0\.15, the all\-row binary\-cap budget, and no external noise cap\. TabArena evaluation uses 594 official outer splits with the benchmark’s internal validation and refitting procedures; TALENT uses five stratified folds per dataset\. BeyondArena follows the official outer splits without inner cross\-validation for TFMs\. Appendix[D](https://arxiv.org/html/2610.12904#A4)specifies tuning, inference, and dataset coverage\.

Environment\.Our TFM runs use PPU\-ZW810E servers with 16 accelerators per node \(95\.6 GiB each\), running Ubuntu 24\.04\.2 LTS, PPU SDK 2\.1\.0, and driver 1\.3\.2\. Pretraining uses Python 3\.12 and a PPU build of PyTorch 2\.9\.0, with FP32 and single\-node distributed data parallelism\.

### 4\.2Predictive performance

We evaluate TabICLv2, TabSwift, and TabPFN\-2\.5 trained with GIST against their Hard counterparts, which use the original GraphSCM generator and sampled hard labels\. We also include five representative baselines: three gradient\-boosted tree methods \(CatBoost, LightGBM, and XGBoost\) and two neural models \(RealMLP and TabM\)\. Evaluation covers the IID splits of TabArena and TALENT and the OOD splits of BeyondArena\. Table[1](https://arxiv.org/html/2610.12904#S4.T1)summarizes IID performance across all methods, while Table[3](https://arxiv.org/html/2610.12904#S4.T3)reports GIST’s paired win rates against Hard within each architecture across the three benchmarks\.

Table 1:IID performance of the complete GIST design\.AUC: binary ROC\-AUC; LL: multiclass log loss;II: Improvability \(%\); Rank: mean rank among 11 methods\. Bold marks the best value per column;blue/greenmarks improvement/deterioration against same\-architecture Hard\.MethodTabArena \(38\)TALENT \(188\)AUC↑\\uparrow\(30\)LL↓\\downarrow\(8\)I↓I\\downarrowRank↓\\downarrowAUC↑\\uparrow\(120\)LL↓\\downarrow\(68\)I↓I\\downarrowRank↓\\downarrowCatBoost0\.84960\.261111\.476\.030\.85360\.427326\.087\.09LightGBM0\.84800\.261412\.526\.200\.86060\.450326\.577\.36XGBoost0\.84730\.261013\.086\.890\.84810\.473528\.797\.61RealMLP0\.84910\.254310\.865\.980\.85930\.457229\.738\.14TabM0\.84880\.261511\.435\.930\.85130\.442228\.587\.70TabICLv2 \(Hard\)0\.84950\.25419\.045\.620\.88040\.397712\.664\.56\+ GIST0\.85010\.25499\.235\.370\.88170\.396711\.624\.09TabSwift \(Hard\)0\.84200\.284815\.758\.110\.87130\.426720\.996\.66\+ GIST0\.84250\.281115\.317\.700\.87260\.424319\.916\.24TabPFN\-2\.5 \(Hard\)0\.84950\.28848\.614\.420\.88630\.39097\.523\.32\+ GIST0\.85290\.25364\.573\.740\.88640\.39157\.773\.23IID prediction\.Across the six combinations of architecture and benchmark, GIST improves mean rank and mean binary ROC\-AUC in all six, and mean multiclass log loss and Improvability in four \(Table[1](https://arxiv.org/html/2610.12904#S4.T1)\)\. It also wins on a majority of datasets in every combination \(Table[3](https://arxiv.org/html/2610.12904#S4.T3)\)\. TabSwift improves all four reported metrics on both IID benchmarks, with paired win rates of 65\.8% on TabArena and 67\.0% on TALENT\. TabPFN\-2\.5 achieves the largest decrease in Improvability on TabArena, from 8\.61% to 4\.57%, with 27/38 paired wins\.

OOD prediction\.On the architecture\-specific BeyondArena task sets, GIST wins on 15/18 datasets for TabSwift, 11/21 for TabICLv2, and 7/12 for TabPFN\-2\.5 \(Table[3](https://arxiv.org/html/2610.12904#S4.T3)\)\. Each comparison includes every task with all official splits completed for both conditions\. On the 12 datasets shared by all methods, GIST lowers mean multiclass log loss for all three architectures; mean binary AUC improves for TabICLv2 and TabSwift but decreases for TabPFN\-2\.5\. Appendix[D\.3](https://arxiv.org/html/2610.12904#A4.SS3)gives the full metrics and comparisons with external baselines\. The results show that GIST can improve prediction under the evaluated distribution shifts, with gains varying across architectures and metrics\.

### 4\.3Gradient variance across training and components

At seven checkpoints along each GIST training trajectory, we compare Hard and selective GIST under the same calibrated GraphSCM prior, sharing weights, inputs, logits, and loss normalization\. We use tasks with tractable conditional query targets at table lengthsL∈\{1024,8192\}L\\in\\\{1024,8192\\\}with 80% context rows and clip each whole\-batch gradient at global norm one\. Figure[3](https://arxiv.org/html/2610.12904#S4.F3)decomposes the percentage reductionRRin gradient covariance trace relative to Hard into component contributionscmc\_\{m\}in percentage points\. Within\-component relative reductions use each component’s own Hard variance as the baseline\. Pointwise 95% bootstrap intervals quantify batch\-sampling variability at fixed weights\. Appendices[D\.4](https://arxiv.org/html/2610.12904#A4.SS4)and[D\.5](https://arxiv.org/html/2610.12904#A4.SS5)give the estimator, decomposition, and parameter partitions\.

Figure 3:Component contributions to clipped\-gradient variance reduction\.Segments show contributionscmc\_\{m\}in percentage points of whole\-model Hard variance; hollow diamonds show their sumRRwith pointwise 95% batch\-pair bootstrap intervals\. Shading marks Stage 2; checkpoints are equally spaced for display, and panels use separate linearyy\-axis ranges\.Table 2:Paired dataset\-level win rates \(%\)\.GIST versus same\-architecture Hard, retaining ties in the denominator\. IID sets match Table[1](https://arxiv.org/html/2610.12904#S4.T1); BeyondArena uses architecture\-specific complete sets \(counts in parentheses\)\.Table 3:TabSwift\-GIST entropy\-budget sensitivity\.Eachβ\\betais fixed across both pretraining stages\. Datasets and metrics follow Table[1](https://arxiv.org/html/2610.12904#S4.T1); bold marks the best mean per column\.ArchitectureTabArenaTALENTBeyondArenaTabICLv255\.364\.452\.4 \(21\)TabSwift65\.867\.083\.3 \(18\)TabPFN\-2\.571\.153\.758\.3 \(12\)
β\\betaTabArenaTALENT \(188\)AUC↑\\uparrowLL↓\\downarrowAUC↑\\uparrowLL↓\\downarrowHard0\.84200\.28480\.87130\.42670\.050\.84270\.28240\.87310\.42500\.150\.84250\.28110\.87260\.42430\.300\.84230\.28220\.87270\.4240

Reduction beyond early training\.The net reductions in Figure[3](https://arxiv.org/html/2610.12904#S4.F3)are below 1% at initialization for all three architectures and both table lengths\. AtL=1024L=1024, TabSwift and TabPFN\-2\.5 already show most of their final reduction by 10K updates; atL=8192L=8192, their largest estimates occur at 36K\. At 36K updates, the reductions range from 6\.80% to 26\.04%, with all six pointwise intervals above zero\. The final reduction is smaller atL=8192L=8192than atL=1024L=1024for every architecture\. The larger effect at trained checkpoints is consistent with the parameter dependence in Equation \([11](https://arxiv.org/html/2610.12904#S3.E11)\): before clipping, the logit Jacobian maps conditional label covariance into parameter space\. Even with fixed conditional targets, changes in this Jacobian can alter the covariance removed by averaging\.

Beyond the prediction head\.At 36K updates, the prediction head has the largest estimated within\-component relative reduction in all six settings, ranging from 37\.70% to 47\.23% \(Appendix Tables[8](https://arxiv.org/html/2610.12904#A4.T8)and[9](https://arxiv.org/html/2610.12904#A4.T9)\)\. The head accounts for only 1\.2%–13\.9% of Hard gradient variance in these settings, so its large proportional response has limited weight in the whole\-model total\. Components outside the head account for 72\.0%–91\.5% of the net whole\-model reduction\. Although GIST changes supervision at the query loss, its measured variance effect extends to the encoders and backbone components\.

Contributions across architectures\.Column\-processing components are major contributors across training in TabICLv2 and TabPFN\-2\.5\. TFcol has the largest estimated contribution at every measured noninitial checkpoint for both table lengths, accounting for more than half of the net reduction in each setting\. Within\-column attention in TabPFN\-2\.5 ranks among the top two contributors at these checkpoints, leading at every checkpoint forL=1024L=1024and from 20K onward forL=8192L=8192\. Both components aggregate information across samples within feature columns or groups\. In TabSwift, MLPs contribute the most in 11 of the 12 settings after initialization, including both final checkpoints\. The leading contributors are thus largely stable across trained checkpoints, but their identity differs across architectures\.

### 4\.4Entropy\-budget sensitivity

The entropy budget controls the residual label uncertainty permitted in the encoding channel, making it a design parameter of the generator–supervision pair\. We pretrain TabSwift withβ∈\{0\.05,0\.15,0\.30\}\\beta\\in\\\{0\.05,0\.15,0\.30\\\}under the same training schedule and evaluation protocol\. All three tested budgets improve mean binary AUC and multiclass log loss over Hard on both IID benchmarks \(Table[3](https://arxiv.org/html/2610.12904#S4.T3)\)\. The preferred value varies by benchmark and metric, but gains extend beyond the default budget ofβ=0\.15\\beta=0\.15to all three tested values\. Appendix[D\.6](https://arxiv.org/html/2610.12904#A4.SS6)gives paired win rates for each budget and the full protocol\.

## 5Related Work

#### Tabular Prediction Tasks\.

Tabular prediction comprises classification and regression from structured records containing numerical and categorical features\. A table can mix continuous measurements, categorical identifiers, and text fields, and column meanings can differ across datasets\. This heterogeneity affects how models represent features and share information across tasks\. Most approaches fit a separate predictor to each dataset\. Gradient\-boosted trees, including XGBoost, LightGBM, and CatBoost, are widely used in this setting\([Chen and Guestrin, 2016](https://arxiv.org/html/2610.12904#bib.bib13);[Ke et al\., 2017](https://arxiv.org/html/2610.12904#bib.bib14);[Prokhorenkova et al\., 2018](https://arxiv.org/html/2610.12904#bib.bib12)\)\. Neural methods learn feature representations and interactions, with transformer and retrieval\-based models providing alternatives\([Gorishniy et al\., 2021](https://arxiv.org/html/2610.12904#bib.bib15);[Gorishniy et al\., 2024](https://arxiv.org/html/2610.12904#bib.bib16)\)\. AutoML systems combine model selection and ensembling to adapt these learners to individual datasets\([Erickson et al\., 2020](https://arxiv.org/html/2610.12904#bib.bib18)\)\. OpenML\-CC18, TabZilla, TALENT, and TabArena support comparisons across datasets with different sample sizes, feature types, and target distributions\([Bischl et al\., 2021](https://arxiv.org/html/2610.12904#bib.bib19);[McElfresh et al\., 2023](https://arxiv.org/html/2610.12904#bib.bib20);[Ye et al\., 2024](https://arxiv.org/html/2610.12904#bib.bib24);[Liu et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib25);[Erickson et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib21)\)\.

#### Tabular Foundation Models\.

TFMs learn across tables during pretraining and use that experience to predict on new datasets\. PFNs implement this idea through synthetic tasks sampled from a prior, whose posterior predictive distribution defines the population target\([Müller et al\., 2022](https://arxiv.org/html/2610.12904#bib.bib28);[Nagler, 2023](https://arxiv.org/html/2610.12904#bib.bib29)\)\. The generator defines the task distribution, while the supervision rule determines how sampled tasks contribute to pretraining\. TabPFN, TabICL, and Mitra develop this approach through synthetic\-prior construction and model design\([Hollmann et al\., 2023](https://arxiv.org/html/2610.12904#bib.bib1);[Hollmann et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib2);[Qu et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib3);[Zhang et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib5)\)\. TabICLv2, TabFlex, and TabSwift further address context scaling and architectural efficiency\([Qu et al\., 2026](https://arxiv.org/html/2610.12904#bib.bib4);[Zeng et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib7);[Liu and Ye, 2026](https://arxiv.org/html/2610.12904#bib.bib8)\)\. TabDPT pretrains on real tables\([Ma et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib22)\), while Real\-TabPFN and Real\-TabPFN\-2\.5 continue pretraining synthetic\-data models on real datasets\([Garg et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib6);[Grinsztajn et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib23)\)\. Semantic models exploit column names and cell text as transferable information\([Kim et al\., 2024](https://arxiv.org/html/2610.12904#bib.bib9);[Spinaci et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib10);[Arazi et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib11)\)\.

#### Generator\-Informed Learning\.

PIQL augments TFMs with dataset statistics and generator\-program representations, using a student module to reconstruct the latter from visible context at inference\([Ding and Akoglu, 2026](https://arxiv.org/html/2610.12904#bib.bib32)\)\. GIST uses generator information to construct query targets while retaining the predictor and inference procedure\. ALICE replaces sampled labels with conditional class probabilities derived from joint likelihood ratios for simulator\-based likelihood\-ratio estimation\([Stoye et al\., 2018](https://arxiv.org/html/2610.12904#bib.bib31)\)\. GIST shares this conditional\-averaging principle and jointly designs task generation and supervision to obtain tractable posterior targets through an entropy\-calibrated local channel in supported label\-dependent branches\.

## 6Conclusion

Synthetic generators can support pretraining through the conditional label information they retain\. GIST makes this information usable by jointly designing task generation and query supervision, enabling conditional loss averaging that preserves the population objective within each fixed generating distribution\. Improvements across three TFM architectures on TabArena and TALENT support the practical value of this design without changes to model architecture or inference\. These results motivate considering the supervision a synthetic generator can provide alongside the tasks it produces, with transfer under distribution shift remaining an open direction\.

### AI use statement

We used generative AI tools to assist with drafting and revising selected parts of the manuscript, including figure and table captions, experimental discussion, and language polishing\. Generative AI was also used to provide feedback on the organization of technical material, experimental analysis, and interpretation of reported results\. The authors reviewed and revised all AI\-assisted content and take full responsibility for the final manuscript\.

## References

- Araziet al\.\(2025\)A\. Arazi, E\. Shapira, and R\. ReichartTabSTAR: A tabular foundation model for tabular data with text fields\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2\-7, 2025 / Mexico City, Mexico, November 30 \- December 5, 2025,D\. Belgrave, C\. Zhang, L\. N\. Montoya, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, N\. Chen, I\. V\. M\. Ruíz, and A\. Loaiza\-Bonilla \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper/_files/paper/2025/hash/faf6e23e198314c7728eaa6ac44ae079-Abstract-Conference.html)Cited by:[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px2.p1.1)\.
- Bischlet al\.\(2021\)B\. Bischl, G\. Casalicchio, M\. Feurer, P\. Gijsbers, F\. Hutter, M\. Lang, R\. G\. G\. Mantovani, J\. N\. van Rijn, and J\. VanschorenOpenML benchmarking suites\.InAdvances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, Datasets and Benchmarks Track,External Links:1708\.03731Cited by:[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px1.p1.1)\.
- Casella and Robert \(1996\)G\. Casella and C\. P\. RobertRao\-Blackwellisation of sampling schemes\.Biometrika83\(1\),pp\. 81–94\.External Links:ISSN 0006\-3444,[Document](https://dx.doi.org/10.1093/biomet/83.1.81),[Link](https://doi.org/10.1093/biomet/83.1.81),https://academic\.oup\.com/biomet/article\-pdf/83/1/81/709586/83\-1\-81\.pdfCited by:[§1](https://arxiv.org/html/2610.12904#S1.p4.1)\.
- Chen and Guestrin \(2016\)T\. Chen and C\. GuestrinXGBoost: A scalable tree boosting system\.InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13\-17, 2016,pp\. 785–794\.External Links:[Link](https://doi.org/10.1145/2939672.2939785),[Document](https://dx.doi.org/10.1145/2939672.2939785)Cited by:[§4\.1](https://arxiv.org/html/2610.12904#S4.SS1.p2.1),[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px1.p1.1)\.
- Ding and Akoglu \(2026\)X\. Ding and L\. AkogluToward privileged foundation models: LUPI for accelerated and improved learning\.Note:arXiv preprint arXiv:2605\.07799External Links:2605\.07799,[Link](https://arxiv.org/abs/2605.07799)Cited by:[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px3.p1.1)\.
- Ericksonet al\.\(2020\)N\. Erickson, J\. Mueller, A\. Shirkov, H\. Zhang, P\. Larroy, M\. Li, and A\. SmolaAutoGluon\-Tabular: robust and accurate automl for structured data\.CoRRabs/2003\.06505\.External Links:2003\.06505Cited by:[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px1.p1.1)\.
- Ericksonet al\.\(2025\)N\. Erickson, L\. Purucker, A\. Tschalzev, D\. Holzmüller, P\. M\. Desai, D\. Salinas, and F\. HutterTabArena: a living benchmark for machine learning on tabular data\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2\-7, 2025, Datasets and Benchmarks Track,External Links:[Document](https://dx.doi.org/10.52202/085713-0519),2506\.16791Cited by:[§D\.2](https://arxiv.org/html/2610.12904#A4.SS2.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2610.12904#S1.p6.1),[§4\.1](https://arxiv.org/html/2610.12904#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.12904#S4.SS1.p3.1),[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px1.p1.1)\.
- Garget al\.\(2025\)A\. Garg, M\. Ali, N\. Hollmann, L\. Purucker, S\. Müller, and F\. HutterReal\-tabpfn: improving tabular foundation models via continued pre\-training with real\-world data\.CoRRabs/2507\.03971\.External Links:[Link](https://doi.org/10.48550/arXiv.2507.03971),[Document](https://dx.doi.org/10.48550/ARXIV.2507.03971),2507\.03971Cited by:[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px2.p1.1)\.
- Gorishniyet al\.\(2025\)Y\. Gorishniy, A\. Kotelnikov, and A\. BabenkoTabM: advancing tabular deep learning with parameter\-efficient ensembling\.InInternational Conference on Learning Representations,Cited by:[§4\.1](https://arxiv.org/html/2610.12904#S4.SS1.p2.1)\.
- Gorishniyet al\.\(2024\)Y\. Gorishniy, I\. Rubachev, N\. Kartashev, D\. Shlenskii, A\. Kotelnikov, and A\. BabenkoTabR: tabular deep learning meets nearest neighbors\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:2307\.14338Cited by:[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px1.p1.1)\.
- Gorishniyet al\.\(2021\)Y\. Gorishniy, I\. Rubachev, V\. Khrulkov, and A\. BabenkoRevisiting deep learning models for tabular data\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px1.p1.1)\.
- Grinsztajnet al\.\(2025\)L\. Grinsztajn, K\. Flöge, O\. Key, F\. Birkel, P\. Jund, B\. Roof, B\. Jäger, D\. Safaric, S\. Alessi, A\. Hayler, M\. Manium, R\. Yu, F\. Jablonski, S\. B\. Hoo, A\. Garg, J\. Robertson, M\. Bühler, V\. Moroshan, L\. Purucker, C\. Cornu, L\. C\. Wehrhahn, A\. Bonetto, B\. Schölkopf, S\. Gambhir, N\. Hollmann, and F\. HutterTabPFN\-2\.5: advancing the state of the art in tabular foundation models\.CoRRabs/2511\.08667\.External Links:[Link](https://doi.org/10.48550/arXiv.2511.08667),[Document](https://dx.doi.org/10.48550/ARXIV.2511.08667),2511\.08667Cited by:[§1](https://arxiv.org/html/2610.12904#S1.p6.1),[§4\.1](https://arxiv.org/html/2610.12904#S4.SS1.p2.1),[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px2.p1.1)\.
- Hollmannet al\.\(2023\)N\. Hollmann, S\. Müller, K\. Eggensperger, and F\. HutterTabPFN: a transformer that solves small tabular classification problems in a second\.InThe Eleventh International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=cp5PvcI6w8_)Cited by:[§1](https://arxiv.org/html/2610.12904#S1.p2.1),[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px2.p1.1)\.
- Hollmannet al\.\(2025\)N\. Hollmann, S\. Müller, L\. Purucker, A\. Krishnakumar, M\. Koerfer, S\. B\. Hoo, R\. T\. Schirrmeister, and F\. HutterAccurate predictions on small data with a tabular foundation model\.Nature637,pp\. 319–326\.External Links:[Document](https://dx.doi.org/10.1038/s41586-024-08328-6)Cited by:[§1](https://arxiv.org/html/2610.12904#S1.p1.1),[§2](https://arxiv.org/html/2610.12904#S2.p1.1),[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px2.p1.1)\.
- Holzmülleret al\.\(2024\)D\. Holzmüller, L\. Grinsztajn, and I\. SteinwartBetter by default: strong pre\-tuned mlps and boosted trees on tabular data\.InAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper/_files/paper/2024/hash/2ee1c87245956e3eaa71aaba5f5753eb-Abstract-Conference.html)Cited by:[§4\.1](https://arxiv.org/html/2610.12904#S4.SS1.p2.1)\.
- Keet al\.\(2017\)G\. Ke, Q\. Meng, T\. Finley, T\. Wang, W\. Chen, W\. Ma, Q\. Ye, and T\. LiuLightGBM: A highly efficient gradient boosting decision tree\.InAdvances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4\-9, 2017, Long Beach, CA, USA,pp\. 3146–3154\.Cited by:[§4\.1](https://arxiv.org/html/2610.12904#S4.SS1.p2.1),[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px1.p1.1)\.
- Kimet al\.\(2024\)M\. J\. Kim, L\. Grinsztajn, and G\. VaroquauxCARTE: pretraining and transfer for tabular learning\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),External Links:2402\.16785Cited by:[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px2.p1.1)\.
- Kolchinsky and Tracey \(2017\)A\. Kolchinsky and B\. D\. TraceyEstimating mixture entropy with pairwise distances\.Entropy19\(7\),pp\. 361\.Note:Corrected version: arXiv:1706\.02419v4 \(2018\)External Links:[Document](https://dx.doi.org/10.3390/e19070361),[Link](https://arxiv.org/abs/1706.02419v4)Cited by:[§C\.1](https://arxiv.org/html/2610.12904#A3.SS1.p1.2),[§3\.3](https://arxiv.org/html/2610.12904#S3.SS3.p2.2)\.
- Liuet al\.\(2025\)S\. Liu, H\. Cai, Q\. Zhou, H\. Yin, T\. Zhou, J\. Jiang, and H\. YeTalent: A tabular analytics and learning toolbox\.J\. Mach\. Learn\. Res\.26,pp\. 226:1–226:16\.External Links:[Link](https://jmlr.org/papers/v26/25-0512.html)Cited by:[§1](https://arxiv.org/html/2610.12904#S1.p6.1),[§4\.1](https://arxiv.org/html/2610.12904#S4.SS1.p1.1),[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px1.p1.1)\.
- Liu and Ye \(2026\)S\. Liu and H\. YeTabSwift: an efficient tabular foundation model with row\-wise attention\.CoRRabs/2606\.07345\.External Links:2606\.07345Cited by:[§1](https://arxiv.org/html/2610.12904#S1.p6.1),[§4\.1](https://arxiv.org/html/2610.12904#S4.SS1.p2.1),[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px2.p1.1)\.
- Maet al\.\(2025\)J\. Ma, V\. Thomas, R\. Hosseinzadeh, A\. Labach, J\. C\. Cresswell, K\. Golestan, G\. Yu, A\. L\. Caterini, and M\. VolkovsTabDPT: scaling tabular foundation models on real data\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2\-7, 2025 / Mexico City, Mexico, November 30 \- December 5, 2025,D\. Belgrave, C\. Zhang, L\. N\. Montoya, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, N\. Chen, I\. V\. M\. Ruíz, and A\. Loaiza\-Bonilla \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper/_files/paper/2025/hash/fc0e3f908a2116ba529ad0a1530a3675-Abstract-Conference.html)Cited by:[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px2.p1.1)\.
- McElfreshet al\.\(2023\)D\. McElfresh, S\. Khandagale, J\. Valverde, V\. P\. C, G\. Ramakrishnan, M\. Goldblum, and C\. WhiteWhen do neural nets outperform boosted trees on tabular data?\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,Cited by:[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px1.p1.1)\.
- Mülleret al\.\(2022\)S\. Müller, N\. Hollmann, S\. P\. Arango, J\. Grabocka, and F\. HutterTransformers can do Bayesian inference\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=KSugKcbNf9)Cited by:[§1](https://arxiv.org/html/2610.12904#S1.p1.1),[§1](https://arxiv.org/html/2610.12904#S1.p3.1),[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px2.p1.1)\.
- Nagler \(2023\)T\. NaglerStatistical foundations of prior\-data fitted networks\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 25660–25676\.External Links:[Link](https://proceedings.mlr.press/v202/nagler23a.html)Cited by:[§A\.6](https://arxiv.org/html/2610.12904#A1.SS6.p1.1),[§1](https://arxiv.org/html/2610.12904#S1.p3.1),[§2](https://arxiv.org/html/2610.12904#S2.p2.1),[§2](https://arxiv.org/html/2610.12904#S2.p3.2),[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px2.p1.1)\.
- Prokhorenkovaet al\.\(2018\)L\. O\. Prokhorenkova, G\. Gusev, A\. Vorobev, A\. V\. Dorogush, and A\. GulinCatBoost: unbiased boosting with categorical features\.InAdvances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, Montréal, Canada, December 3\-8, 2018,pp\. 6639–6649\.Cited by:[§4\.1](https://arxiv.org/html/2610.12904#S4.SS1.p2.1),[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px1.p1.1)\.
- Puruckeret al\.\(2026\)L\. Purucker, A\. Tschalzev, N\. Erickson, G\. Blayer, D\. Holzmüller, A\. Arazi, A\. Pfefferle, M\. Tajjar, G\. Varoquaux, and F\. HutterBeyond IID: how general are tabular foundation models, really?\.CoRRabs/2606\.30410\.External Links:2606\.30410,[Link](https://arxiv.org/abs/2606.30410)Cited by:[§1](https://arxiv.org/html/2610.12904#S1.p6.1),[§4\.1](https://arxiv.org/html/2610.12904#S4.SS1.p1.1)\.
- Quet al\.\(2025\)J\. Qu, D\. Holzmüller, G\. Varoquaux, and M\. L\. MorvanTabICL: a tabular foundation model for in\-context learning on large data\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),External Links:2502\.05564Cited by:[§1](https://arxiv.org/html/2610.12904#S1.p1.1),[§1](https://arxiv.org/html/2610.12904#S1.p2.1),[§2](https://arxiv.org/html/2610.12904#S2.p1.1),[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px2.p1.1)\.
- Quet al\.\(2026\)J\. Qu, D\. Holzmüller, G\. Varoquaux, and M\. L\. MorvanTabICLv2: a better, faster, scalable, and open tabular foundation model\.CoRRabs/2602\.11139\.External Links:2602\.11139Cited by:[§B\.4](https://arxiv.org/html/2610.12904#A2.SS4.p1.1),[§1](https://arxiv.org/html/2610.12904#S1.p2.1),[§1](https://arxiv.org/html/2610.12904#S1.p6.1),[§4\.1](https://arxiv.org/html/2610.12904#S4.SS1.p2.1),[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px2.p1.1)\.
- Spinaciet al\.\(2025\)M\. Spinaci, M\. Polewczyk, M\. Schambach, and S\. ThelinConTextTab: A semantics\-aware tabular in\-context learner\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2\-7, 2025 / Mexico City, Mexico, November 30 \- December 5, 2025,D\. Belgrave, C\. Zhang, L\. N\. Montoya, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, N\. Chen, I\. V\. M\. Ruíz, and A\. Loaiza\-Bonilla \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper/_files/paper/2025/hash/d807e7678ba3afd3a904f4af52819e77-Abstract-Conference.html)Cited by:[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px2.p1.1)\.
- Stoyeet al\.\(2018\)M\. Stoye, J\. Brehmer, G\. Louppe, J\. Pavez, and K\. CranmerLikelihood\-free inference with an improved cross\-entropy estimator\.Note:arXiv preprint arXiv:1808\.00973External Links:1808\.00973,[Link](https://arxiv.org/abs/1808.00973)Cited by:[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px3.p1.1)\.
- Yeet al\.\(2024\)H\. Ye, S\. Liu, H\. Cai, Q\. Zhou, and D\. ZhanA closer look at deep learning methods on tabular datasets\.CoRRabs/2407\.00956\.External Links:[Link](https://doi.org/10.48550/arXiv.2407.00956),[Document](https://dx.doi.org/10.48550/ARXIV.2407.00956),2407\.00956Cited by:[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px1.p1.1)\.
- Zenget al\.\(2025\)Y\. Zeng, T\. Dinh, W\. Kang, and A\. C\. MuellerTabFlex: scaling tabular learning to millions with linear attention\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.267,pp\. 74051–74079\.External Links:[Link](https://proceedings.mlr.press/v267/zeng25b.html)Cited by:[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2025\)X\. Zhang, D\. C\. Maddix, J\. Yin, N\. Erickson, A\. F\. Ansari, B\. Han, S\. Zhang, L\. Akoglu, C\. Faloutsos, M\. W\. Mahoney, C\. Hu, H\. Rangwala, G\. Karypis, and B\. WangMitra: mixed synthetic priors for enhancing tabular foundation models\.CoRRabs/2510\.21204\.External Links:[Link](https://doi.org/10.48550/arXiv.2510.21204),[Document](https://dx.doi.org/10.48550/ARXIV.2510.21204),2510\.21204Cited by:[§1](https://arxiv.org/html/2610.12904#S1.p2.1),[§5](https://arxiv.org/html/2610.12904#S5.SS0.SSS0.Px2.p1.1)\.

## Appendix AConditional Supervision: Proofs and Extensions

### A\.1Regularity conditions and objective preservation

Use the lawPPfrom Section[3\.4](https://arxiv.org/html/2610.12904#S3.SS4), withSSmeasurable with respect toZZ\. Assumeqθ​\(k∣S\)\>0q\_\{\\theta\}\(k\\mid S\)\>0for classes with positive conditional probability\. The losses are integrable, their gradients have finite second moments, and differentiation can be interchanged with the expectations used below\. One sufficient local condition for the last requirement is differentiability together with an integrable bound on the gradient in a neighborhood ofθ\\theta\. Model randomness is held fixed; alternatively, independent model randomness can be included in bothSSandZZ\.

Sinceqθ\(⋅∣S\)q\_\{\\theta\}\(\\cdot\\mid S\)is fixed conditional onZZ,

𝔼⁡\[ℓH∣Z\]\\displaystyle\\mathbb\{E\}\[\\ell\_\{\\mathrm\{H\}\}\\mid Z\]=−∑k=1KP\(Y=k∣Z\)logqθ\(k∣S\)=ℓRB,\\displaystyle=\-\\sum\_\{k=1\}^\{K\}P\(Y=k\\mid Z\)\\log q\_\{\\theta\}\(k\\mid S\)=\\ell\_\{\\mathrm\{RB\}\},\(12\)𝔼⁡\[ℓRB\]\\displaystyle\\mathbb\{E\}\[\\ell\_\{\\mathrm\{RB\}\}\]=𝔼⁡\[𝔼⁡\[ℓH∣Z\]\]=R⁡\(θ\)\.\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\mathbb\{E\}\[\\ell\_\{\\mathrm\{H\}\}\\mid Z\]\\right\]=R\(\\theta\)\.To connect this equality to the model’s prediction target, defineℓP\(θ;S\)=−∑kπS\(k\)logqθ\(k∣S\)\\ell\_\{\\mathrm\{P\}\}\(\\theta;S\)=\-\\sum\_\{k\}\\pi\_\{S\}\(k\)\\log q\_\{\\theta\}\(k\\mid S\)\. The tower property gives

𝔼⁡\[ℓRB∣S\]=ℓP,𝔼⁡\[ℓP\]=R⁡\(θ\)\.\\mathbb\{E\}\[\\ell\_\{\\mathrm\{RB\}\}\\mid S\]=\\ell\_\{\\mathrm\{P\}\},\\qquad\\mathbb\{E\}\[\\ell\_\{\\mathrm\{P\}\}\]=R\(\\theta\)\.\(13\)Thus conditioning the supervision on generator information preserves the objective associated with the posterior predictive distribution\. The generator suppliespZp\_\{Z\}during training, while the predictor continues to estimateπS\\pi\_\{S\}fromSS\.

### A\.2Gradient covariance and the softmax expression

Differentiating the conditional\-loss identity yieldsGRB=𝔼⁡\[GH∣Z\]G\_\{\\mathrm\{RB\}\}=\\mathbb\{E\}\[G\_\{\\mathrm\{H\}\}\\mid Z\]\. LetΔ=GH−GRB\\Delta=G\_\{\\mathrm\{H\}\}\-G\_\{\\mathrm\{RB\}\}\. Then𝔼⁡\[Δ∣Z\]=0\\mathbb\{E\}\[\\Delta\\mid Z\]=0, so its cross\-covariance with any square\-integrable function ofZZ, includingGRBG\_\{\\mathrm\{RB\}\}, vanishes\. Consequently,

VH−VRB=𝔼⁡\[Δ​Δ⊤\]=𝔼⁡\[Γθ​\(Z\)\]⪰0\.V\_\{\\mathrm\{H\}\}\-V\_\{\\mathrm\{RB\}\}=\\mathbb\{E\}\[\\Delta\\Delta^\{\\top\}\]=\\mathbb\{E\}\[\\Gamma\_\{\\theta\}\(Z\)\]\\succeq 0\.\(14\)For any fixed vectorvv, the quadratic form of this difference is𝔼⁡\[\(v⊤​Δ\)2\]≥0\\mathbb\{E\}\[\(v^\{\\top\}\\Delta\)^\{2\}\]\\geq 0\. This proves the covariance statement in Proposition[1](https://arxiv.org/html/2610.12904#Thmproposition1)\. Since both gradients have mean∇R​\(θ\)\\nabla R\(\\theta\), taking traces also gives an exact mean\-squared error reduction:

𝔼​‖GH−∇R​\(θ\)‖22−𝔼​‖GRB−∇R​\(θ\)‖22\\displaystyle\\mathbb\{E\}\\\|G\_\{\\mathrm\{H\}\}\-\\nabla R\(\\theta\)\\\|\_\{2\}^\{2\}\-\\mathbb\{E\}\\\|G\_\{\\mathrm\{RB\}\}\-\\nabla R\(\\theta\)\\\|\_\{2\}^\{2\}\(15\)=𝔼​‖GH−GRB‖22=𝔼⁡\[tr⁡Γθ​\(Z\)\]≥0\.\\displaystyle=\\mathbb\{E\}\\\|G\_\{\\mathrm\{H\}\}\-G\_\{\\mathrm\{RB\}\}\\\|\_\{2\}^\{2\}=\\mathbb\{E\}\\\!\\left\[\\operatorname\{tr\}\\Gamma\_\{\\theta\}\(Z\)\\right\]\\geq 0\.The conditional average therefore uses the same prediction to estimate the population gradient with no larger mean\-squared error\.

For softmax logitsaθa\_\{\\theta\}, letJθ∈ℝK×dθJ\_\{\\theta\}\\in\\mathbb\{R\}^\{K\\times d\_\{\\theta\}\}be their Jacobian and leteYe\_\{Y\}denote the one\-hot label\. At a fixedSS,

GH=Jθ⊤​\(qθ−eY\),GRB=Jθ⊤​\(qθ−pZ\)\.G\_\{\\mathrm\{H\}\}=J\_\{\\theta\}^\{\\top\}\(q\_\{\\theta\}\-e\_\{Y\}\),\\qquad G\_\{\\mathrm\{RB\}\}=J\_\{\\theta\}^\{\\top\}\(q\_\{\\theta\}\-p\_\{Z\}\)\.\(16\)BecauseCov⁡\(eY∣Z\)=Diag⁡\(pZ\)−pZ​pZ⊤\\operatorname\{Cov\}\(e\_\{Y\}\\mid Z\)=\\operatorname\{Diag\}\(p\_\{Z\}\)\-p\_\{Z\}p\_\{Z\}^\{\\top\}, Equation \([11](https://arxiv.org/html/2610.12904#S3.E11)\) follows\. The trace of this label covariance is1−‖pZ‖221\-\\\|p\_\{Z\}\\\|\_\{2\}^\{2\}\. Its effect in parameter space also depends onJθJ\_\{\\theta\}, and can vanish in directions to which the logits are insensitive\.

### A\.3Remaining sources of gradient variation

For the special caseZ=\(T,S\)Z=\(T,S\), defineGP=∇θℓP=𝔼⁡\[GRB∣S\]G\_\{\\mathrm\{P\}\}=\\nabla\_\{\\theta\}\\ell\_\{\\mathrm\{P\}\}=\\mathbb\{E\}\[G\_\{\\mathrm\{RB\}\}\\mid S\]\. Applying total covariance first conditional onZZand then onSSgives

VH\\displaystyle V\_\{\\mathrm\{H\}\}=Vepisode\+Vtask\+Vlabel,\\displaystyle=V\_\{\\mathrm\{episode\}\}\+V\_\{\\mathrm\{task\}\}\+V\_\{\\mathrm\{label\}\},\(17\)Vepisode\\displaystyle V\_\{\\mathrm\{episode\}\}:=Cov⁡\(GP\),\\displaystyle:=\\operatorname\{Cov\}\(G\_\{\\mathrm\{P\}\}\),Vtask\\displaystyle V\_\{\\mathrm\{task\}\}:=𝔼⁡\[Cov⁡\(GRB∣S\)\],\\displaystyle:=\\mathbb\{E\}\[\\operatorname\{Cov\}\(G\_\{\\mathrm\{RB\}\}\\mid S\)\],Vlabel\\displaystyle V\_\{\\mathrm\{label\}\}:=𝔼⁡\[Γθ​\(Z\)\]\.\\displaystyle:=\\mathbb\{E\}\[\\Gamma\_\{\\theta\}\(Z\)\]\.Conditional supervision removesVlabelV\_\{\\mathrm\{label\}\}\. The termVtaskV\_\{\\mathrm\{task\}\}measures variation across mechanisms compatible with the same visible input, whileVepisodeV\_\{\\mathrm\{episode\}\}remains when the visible input changes\. In particular,VH⪰VRB⪰Cov⁡\(GP\)V\_\{\\mathrm\{H\}\}\\succeq V\_\{\\mathrm\{RB\}\}\\succeq\\operatorname\{Cov\}\(G\_\{\\mathrm\{P\}\}\)\. For a more generalZZ, the same algebra holds, but the middle term describes variation in all retained generator information conditional onSS\.

### A\.4Selective conditional supervision

LetA=A⁡\(Z\)∈\{0,1\}A=A\(Z\)\\in\\\{0,1\\\}be independent ofθ\\theta\. WithGM=A​GRB\+\(1−A\)​GHG\_\{\\mathrm\{M\}\}=AG\_\{\\mathrm\{RB\}\}\+\(1\-A\)G\_\{\\mathrm\{H\}\}, conditioning onZZgives

𝔼⁡\[ℓM∣Z\]\\displaystyle\\mathbb\{E\}\[\\ell\_\{\\mathrm\{M\}\}\\mid Z\]=ℓRB,\\displaystyle=\\ell\_\{\\mathrm\{RB\}\},\(18\)𝔼⁡\[GM∣Z\]\\displaystyle\\mathbb\{E\}\[G\_\{\\mathrm\{M\}\}\\mid Z\]=GRB,\\displaystyle=G\_\{\\mathrm\{RB\}\},Cov⁡\(GM∣Z\)\\displaystyle\\operatorname\{Cov\}\(G\_\{\\mathrm\{M\}\}\\mid Z\)=\(1−A\)​Γθ​\(Z\)\.\\displaystyle=\(1\-A\)\\Gamma\_\{\\theta\}\(Z\)\.The first identity preserves the population objective\. The other two and total covariance imply

VM=VRB\+𝔼⁡\[\(1−A\)​Γθ​\(Z\)\],VH−VM=𝔼⁡\[A​Γθ​\(Z\)\]\.V\_\{\\mathrm\{M\}\}=V\_\{\\mathrm\{RB\}\}\+\\mathbb\{E\}\[\(1\-A\)\\Gamma\_\{\\theta\}\(Z\)\],\\qquad V\_\{\\mathrm\{H\}\}\-V\_\{\\mathrm\{M\}\}=\\mathbb\{E\}\[A\\Gamma\_\{\\theta\}\(Z\)\]\.\(19\)The conditional expectation is defined on every branch, while the implementation evaluates it only whenA=1A=1\. The branch indicator is determined by retained state before the residual label draw, as required for the conditional identities above\.

### A\.5Multiple queries and independent episodes

For a table withQQqueries, letSSinclude the full model input and writeqθ,rq\_\{\\theta,r\}for the prediction at queryrr\. Retain a common information setZZcontainingSS, and definepZ,r​\(k\)=P⁡\(Yr=k∣Z\)p\_\{Z,r\}\(k\)=P\(Y\_\{r\}=k\\mid Z\)\. Linearity of conditional expectation gives

ℓHtable\\displaystyle\\ell\_\{\\mathrm\{H\}\}^\{\\mathrm\{table\}\}:=−1Q∑r=1Qlogqθ,r\(Yr∣S\),\\displaystyle:=\-\\frac\{1\}\{Q\}\\sum\_\{r=1\}^\{Q\}\\log q\_\{\\theta,r\}\(Y\_\{r\}\\mid S\),\(20\)𝔼⁡\[ℓHtable∣Z\]\\displaystyle\\mathbb\{E\}\[\\ell\_\{\\mathrm\{H\}\}^\{\\mathrm\{table\}\}\\mid Z\]=−1Q∑r=1Q∑k=1KpZ,r\(k\)logqθ,r\(k∣S\)\.\\displaystyle=\-\\frac\{1\}\{Q\}\\sum\_\{r=1\}^\{Q\}\\sum\_\{k=1\}^\{K\}p\_\{Z,r\}\(k\)\\log q\_\{\\theta,r\}\(k\\mid S\)\.Hence Proposition[1](https://arxiv.org/html/2610.12904#Thmproposition1)applies to the whole table without assuming conditional independence between its labels\. With row JacobiansJθ,rJ\_\{\\theta,r\}, the corresponding conditional covariance is

Γθtable​\(Z\)=1Q2​∑r=1Q∑s=1QJθ,r⊤​Cov⁡\(eYr,eYs∣Z\)​Jθ,s\.\\Gamma\_\{\\theta\}^\{\\mathrm\{table\}\}\(Z\)=\\frac\{1\}\{Q^\{2\}\}\\sum\_\{r=1\}^\{Q\}\\sum\_\{s=1\}^\{Q\}J\_\{\\theta,r\}^\{\\top\}\\operatorname\{Cov\}\(e\_\{Y\_\{r\}\},e\_\{Y\_\{s\}\}\\mid Z\)J\_\{\\theta,s\}\.\(21\)The off\-diagonal terms vanish under conditional independence\. Averagingmmindependent table gradients divides their covariance bymm\. Within a table, Equation \([21](https://arxiv.org/html/2610.12904#A1.E21)\) accounts for both diagonal and cross\-query contributions\.

### A\.6Asymptotic parameter covariance

We relate the fixed\-parameter comparison to the finite\-dimensional M\-estimation setting of Theorem B\.1 in the PFN analysis\([Nagler, 2023](https://arxiv.org/html/2610.12904#bib.bib29)\)\. Assume a fixed parameter dimension, a unique interior minimizerθ⋆\\theta^\{\\star\}, and a positive\-definite population HessianH⋆=∇2R​\(θ⋆\)H\_\{\\star\}=\\nabla^\{2\}R\(\\theta^\{\\star\}\)\. Fora∈\{H,RB\}a\\in\\\{\\mathrm\{H\},\\mathrm\{RB\}\\\}, letθ^a,m\\widehat\{\\theta\}\_\{a,m\}be an empirical minimizer based onmmindependent episodes\. Assume these minimizers are consistent, are interior with probability tending to one, and have empirical Hessians converging locally uniformly to the continuous population Hessian\. Together with the moment and differentiation conditions above, a Taylor expansion of each empirical score gives

m\(θ^a,m−θ⋆\)=−H⋆−11m∑j=1mGa\(j\)\(θ⋆\)\+oP\(1\)\.\\sqrt\{m\}\(\\widehat\{\\theta\}\_\{a,m\}\-\\theta^\{\\star\}\)=\-H\_\{\\star\}^\{\-1\}\\frac\{1\}\{\\sqrt\{m\}\}\\sum\_\{j=1\}^\{m\}G\_\{a\}^\{\(j\)\}\(\\theta^\{\\star\}\)\+o\_\{P\}\(1\)\.\(22\)The central limit theorem yields

m​\(θ^a,m−θ⋆\)⇒𝒩⁡\(0,Σa\),Σa=H⋆−1​Va​\(θ⋆\)​H⋆−⁣⊤\.\\sqrt\{m\}\(\\widehat\{\\theta\}\_\{a,m\}\-\\theta^\{\\star\}\)\\ \\Rightarrow\\ \\mathcal\{N\}\(0,\\Sigma\_\{a\}\),\\qquad\\Sigma\_\{a\}=H\_\{\\star\}^\{\-1\}V\_\{a\}\(\\theta^\{\\star\}\)H\_\{\\star\}^\{\-\\top\}\.\(23\)The two objectives have the same population Hessian\. Applying Equation \([14](https://arxiv.org/html/2610.12904#A1.E14)\) atθ⋆\\theta^\{\\star\}therefore gives

ΣH−ΣRB=H⋆−1​𝔼​\[Γθ⋆​\(Z\)\]​H⋆−⁣⊤⪰0\.\\Sigma\_\{\\mathrm\{H\}\}\-\\Sigma\_\{\\mathrm\{RB\}\}=H\_\{\\star\}^\{\-1\}\\mathbb\{E\}\[\\Gamma\_\{\\theta^\{\\star\}\}\(Z\)\]H\_\{\\star\}^\{\-\\top\}\\succeq 0\.\(24\)Under these M\-estimation assumptions, the empirical minimizers share them−1/2m^\{\-1/2\}rate, and conditional supervision reduces the covariance of the limiting parameter distribution\.

## Appendix BConditional Targets in the Generator

### B\.1Observed features and task acceptance

Letℋ0\\mathcal\{H\}\_\{0\}denote retained state before a query\-label draw, withpk0=P⁡\(Y=k∣ℋ0=h\)p^\{0\}\_\{k\}=P\(Y=k\\mid\\mathcal\{H\}\_\{0\}=h\), and letE∈\{0,1\}E\\in\\\{0,1\\\}indicate task acceptance\. WritingLk​\(h,s\)L\_\{k\}\(h,s\)for the conditional likelihood ofS=s,E=1S=s,E=1givenℋ0=h,Y=k\\mathcal\{H\}\_\{0\}=h,Y=k, Bayes’ rule gives

P⁡\(Y=k∣ℋ0=h,S=s,E=1\)=pk0​Lk​\(h,s\)∑jpj0​Lj​\(h,s\)\.P\(Y=k\\mid\\mathcal\{H\}\_\{0\}=h,S=s,E=1\)=\\frac\{p^\{0\}\_\{k\}L\_\{k\}\(h,s\)\}\{\\sum\_\{j\}p^\{0\}\_\{j\}L\_\{j\}\(h,s\)\}\.\(25\)Thus the probabilities before sampling remain valid if the observed input and acceptance have a likelihood independent of the label on its support\. This condition holds for a terminal draw when input generation and acceptance no longer read that draw\. In the terminal construction, context\-based class boundaries and label checks are part of the generator shared by hard\-label and RB training\.

When features depend on the sampled label, the likelihood term can change the target\. For example, ifYYis uniform on\{0,1\}\\\{0,1\\\}andX=YX=Y, observingX=1X=1makes the conditional label deterministic\. The likelihood then updates the pre\-draw probabilities\(1/2,1/2\)\(1/2,1/2\)to a point mass at the observed class\. This example separates the probabilities used to generate a label from the probabilities conditional on the final model input\. The retained state includes that input and leaves the label randomness to be averaged unobserved; storing a random seed that determines the query label would instead fix its value\.

### B\.2Posterior under the Gaussian channel

We first use the channel law before filtering and then condition on acceptance\. Fixℋ=h\\mathcal\{H\}=h, including the codebook, all row probabilities, and a scaleσ⁡\(h\)\>0\\sigma\(h\)\>0determined solely by this pre\-draw state\. Thus the scale is measurable with respect toℋ\\mathcal\{H\}, irrespective of when the deterministic calibration routine is executed\. Letϕσ​\(w−b\)\\phi\_\{\\sigma\}\(w\-b\)be the density of𝒩⁡\(b,σ2​Id\)\\mathcal\{N\}\(b,\\sigma^\{2\}I\_\{d\}\)\. Conditional independence of row labels and independent encoding noise imply the joint density

fh\(y1:L,w1:L\)=∏r=1Lpr,yrϕσ\(wr−byr\),f\_\{h\}\(y\_\{1:L\},w\_\{1:L\}\)=\\prod\_\{r=1\}^\{L\}p\_\{r,y\_\{r\}\}\\phi\_\{\\sigma\}\(w\_\{r\}\-b\_\{y\_\{r\}\}\),\(26\)wherefhf\_\{h\}is the joint density with respect to counting measure for labels and Lebesgue measure for encodings\. Conditioning on all encodings and the hard context labels gives

P\(YN\+1:L=yN\+1:L∣ℋ,W1:L,Y1:N\)=∏r=N\+1Lpr,yr​ϕσ​\(Wr−byr\)∑jpr​j​ϕσ​\(Wr−bj\)\.P\(Y\_\{N\+1:L\}=y\_\{N\+1:L\}\\mid\\mathcal\{H\},W\_\{1:L\},Y\_\{1:N\}\)=\\prod\_\{r=N\+1\}^\{L\}\\frac\{p\_\{r,y\_\{r\}\}\\phi\_\{\\sigma\}\(W\_\{r\}\-b\_\{y\_\{r\}\}\)\}\{\\sum\_\{j\}p\_\{rj\}\\phi\_\{\\sigma\}\(W\_\{r\}\-b\_\{j\}\)\}\.\(27\)
The required downstream condition is

YN\+1:L⟂\(S,E\)∣\(ℋ,W1:L,Y1:N\)\.Y\_\{N\+1:L\}\\ \\perp\\ \(S,E\)\\ \\mid\\ \(\\mathcal\{H\},W\_\{1:L\},Y\_\{1:N\}\)\.\(28\)This condition holds when downstream inputs and acceptance are computed from the conditioning variables and auxiliary randomness that is conditionally independent of the query labels\. To see the computational consequence, fix\(h,w1:L,y1:N\)\(h,w\_\{1:L\},y\_\{1:N\}\)and denote the likelihood ofS=s,E=1S=s,E=1byΛ\(s,1∣h,w1:L,y1:N\)\\Lambda\(s,1\\mid h,w\_\{1:L\},y\_\{1:N\}\)\. By Equation \([28](https://arxiv.org/html/2610.12904#A2.E28)\), this likelihood is the same for every candidate query\-label vector\. Multiplying Equation \([27](https://arxiv.org/html/2610.12904#A2.E27)\) byΛ\\Lambdaand normalizing therefore cancels the entire downstream likelihood\. WithZσ=\(ℋ,W1:L,Y1:N,S\)Z\_\{\\sigma\}=\(\\mathcal\{H\},W\_\{1:L\},Y\_\{1:N\},S\), Equation \([28](https://arxiv.org/html/2610.12904#A2.E28)\) implies, on accepted tasks,

P⁡\(Yr=k∣Zσ,E=1\)=P⁡\(Yr=k∣ℋ,Wr\)=p~σ,r​\(k\)\.P\(Y\_\{r\}=k\\mid Z\_\{\\sigma\},E=1\)=P\(Y\_\{r\}=k\\mid\\mathcal\{H\},W\_\{r\}\)=\\widetilde\{p\}\_\{\\sigma,r\}\(k\)\.\(29\)The Gaussian normalizing factors cancel because all classes shareσ\\sigma, giving Equation \([5](https://arxiv.org/html/2610.12904#S3.E5)\)\. Conditioning onSSand acceptance also preserves the product posterior in Equation \([27](https://arxiv.org/html/2610.12904#A2.E27)\)\.

The target uses the likelihood of the actual channel\. For a scalar encoding broadcast into several feature positions, this likelihood is evaluated on the single noisy scalar before broadcasting; repeated copies are not independent observations\. Posterior weights can be computed in log space fromlog⁡pr​k−‖Wr−bk‖22/\(2​σ2\)\\log p\_\{rk\}\-\\\|W\_\{r\}\-b\_\{k\}\\\|\_\{2\}^\{2\}/\(2\\sigma^\{2\}\)and normalized using log\-sum\-exp\. Classes with zero prior probability retain zero posterior probability\.

### B\.3Objective preservation after calibration and fallback

Fix a calibration rule and a decision to enable the channel, both based on state available before the query\-label and encoding\-noise draws\. These choices, the original branches retained as fallback, and the acceptance procedure definePcalP\_\{\\mathrm\{cal\}\}, an instance of the lawPPin Section[3\.4](https://arxiv.org/html/2610.12904#S3.SS4)\. All conditional distributions in this paragraph are taken under that law\.

On enabled branches satisfying Equation \([28](https://arxiv.org/html/2610.12904#A2.E28)\), Equation \([29](https://arxiv.org/html/2610.12904#A2.E29)\) supplies the exact query marginals\. Linearity of conditional expectation then gives

𝔼cal\[−1Q∑r=N\+1Llogqθ,r\(Yr∣S\)\|Zσ\]=−1Q∑r=N\+1L∑kp~σ,r\(k\)logqθ,r\(k∣S\)\.\\mathbb\{E\}\_\{\\mathrm\{cal\}\}\\\!\\left\[\-\\frac\{1\}\{Q\}\\sum\_\{r=N\+1\}^\{L\}\\log q\_\{\\theta,r\}\(Y\_\{r\}\\mid S\)\\,\\middle\|\\,Z\_\{\\sigma\}\\right\]=\-\\frac\{1\}\{Q\}\\sum\_\{r=N\+1\}^\{L\}\\sum\_\{k\}\\widetilde\{p\}\_\{\\sigma,r\}\(k\)\\log q\_\{\\theta,r\}\(k\\mid S\)\.\(30\)The generator state used to combine branches includes the branch decision\. Applying Proposition[1](https://arxiv.org/html/2610.12904#Thmproposition1)on exact conditional\-target branches and Equation \([10](https://arxiv.org/html/2610.12904#S3.E10)\) for hard\-label fallback gives

Rcal,H​\(θ\)=Rcal,RB​\(θ\),Vcal,H−Vcal,RB⪰0,R\_\{\\mathrm\{cal,H\}\}\(\\theta\)=R\_\{\\mathrm\{cal,RB\}\}\(\\theta\),\\qquad V\_\{\\mathrm\{cal,H\}\}\-V\_\{\\mathrm\{cal,RB\}\}\\succeq 0,\(31\)Here the RB procedure includes hard\-label fallback on untreated branches, and both estimators target the population objective ofPcalP\_\{\\mathrm\{cal\}\}\.

### B\.4Instantiation in GraphSCM converters

We apply the preceding rules to the categorical converters of the TabICLv2 generator\([Qu et al\., 2026](https://arxiv.org/html/2610.12904#bib.bib4)\)\. In the softmax converters, a continuous node state determines class probabilities, from which a discrete class is sampled\. The class and the state passed to downstream nodes are separate outputs of the converter\. Thesoftmax\_idmode passes on the original continuous state, so the sampled class does not enter subsequent feature computations\. With query\-label\-independent acceptance, its pre\-draw probabilities provide terminal conditional targets even when the node has observed descendants\.

Thesoftmax\_discmode passes on the sampled class’s vector code, andsoftmax\_intpasses on its scalar integer code\. On these modes, Equation \([4](https://arxiv.org/html/2610.12904#S3.E4)\) supplies the state used by the downstream graph\. When the sampled class can reach an observed feature, supervision uses the posterior in Equation \([5](https://arxiv.org/html/2610.12904#S3.E5)\)\. When the final input and acceptance are independent of the query\-category draw conditional on the pre\-draw state and context labels, the encoding can be omitted from the retained information and the pre\-draw probabilities remain available\. An explicitly infeasible entropy budget selects the sampled hard target as described in Appendix[C\.2](https://arxiv.org/html/2610.12904#A3.SS2)\. Nearest\-neighbor converters assign classes deterministically given the node state and centers, and retain one\-hot targets\.

The same Gaussian scale is used for encoding and posterior evaluation\. A scalar code receives one noise draw per row before broadcasting, and class permutations are applied to both labels and probability columns\. The context supplied to the model keeps its sampled hard labels throughout\.

## Appendix CEntropy Bound and Scale Calibration

### C\.1From mixture entropy to conditional label entropy

Under the channel distribution before task filtering, fix the stateℋ=h\\mathcal\{H\}=h, a rowrr, and a positive scaleσ\\sigma\. Usehdiffh\_\{\\mathrm\{diff\}\}for differential entropy and writehσ=d2​log⁡\(2​π​e​σ2\)h\_\{\\sigma\}=\\tfrac\{d\}\{2\}\\log\(2\\pi e\\sigma^\{2\}\)for the entropy of each Gaussian component\. The discrete\-continuous entropy identity gives

ℰr​\(σ\)=H⁡\(pr\)\+hσ−hdiff​\(Wr∣ℋ=h\)\.\\mathcal\{E\}\_\{r\}\(\\sigma\)=H\(p\_\{r\}\)\+h\_\{\\sigma\}\-h\_\{\\mathrm\{diff\}\}\(W\_\{r\}\\mid\\mathcal\{H\}=h\)\.\(32\)For the mixture∑ipr​i​𝒩​\(bi,σ2​Id\)\\sum\_\{i\}p\_\{ri\}\\mathcal\{N\}\(b\_\{i\},\\sigma^\{2\}I\_\{d\}\), Equation \(12\) in the corrected arXiv version of the mixture\-entropy analysis\([Kolchinsky and Tracey, 2017](https://arxiv.org/html/2610.12904#bib.bib30)\)gives the Bhattacharyya lower bound

hdiff​\(Wr∣ℋ=h\)≥hσ−∑ipr​i​log⁡\[∑jpr​j​exp⁡\(−‖bi−bj‖228​σ2\)\]\.h\_\{\\mathrm\{diff\}\}\(W\_\{r\}\\mid\\mathcal\{H\}=h\)\\geq h\_\{\\sigma\}\-\\sum\_\{i\}p\_\{ri\}\\log\\\!\\left\[\\sum\_\{j\}p\_\{rj\}\\exp\\\!\\left\(\-\\frac\{\\\|b\_\{i\}\-b\_\{j\}\\\|\_\{2\}^\{2\}\}\{8\\sigma^\{2\}\}\\right\)\\right\]\.\(33\)Section IV of that version gives the Gaussian pairwise distances\. Substitution into Equation \([32](https://arxiv.org/html/2610.12904#A3.E32)\) provesℰr​\(σ\)≤Ur​\(σ\)\\mathcal\{E\}\_\{r\}\(\\sigma\)\\leq U\_\{r\}\(\\sigma\)\.

For everyiiwithpr​i\>0p\_\{ri\}\>0, the inner sum lies in\[pr​i,1\]\[p\_\{ri\},1\]\. Its logarithm therefore lies in\[log⁡pr​i,0\]\[\\log p\_\{ri\},0\], which proves0≤Ur​\(σ\)≤H⁡\(pr\)0\\leq U\_\{r\}\(\\sigma\)\\leq H\(p\_\{r\}\)\. Together with the nonnegativity of conditional label entropy, this yields Equation \([6](https://arxiv.org/html/2610.12904#S3.E6)\)\. Zero\-probability terms are interpreted by continuity\.

### C\.2Monotonicity, duplicate codes, and feasibility

Each factorexp\[−∥bi−bj∥22/\(8s2\)\]\\exp\[\-\\\|b\_\{i\}\-b\_\{j\}\\\|\_\{2\}^\{2\}/\(8s^\{2\}\)\]is continuous and nondecreasing fors\>0s\>0\. Positive weighted sums and logarithms preserve this ordering, soUr​\(s\)U\_\{r\}\(s\)andU¯​\(s\)\\overline\{U\}\(s\)are continuous and nondecreasing on that domain\. Their limits at zero exist\. For a code shared by several classes, letρr​i=∑j:bj=bipr​j\\rho\_\{ri\}=\\sum\_\{j:b\_\{j\}=b\_\{i\}\}p\_\{rj\}\. Then

Ur\(0\+\)=H\(pr\)\+∑i:pr​i\>0pr​ilogρr​i=H\(Yr∣bYr,ℋ=h\)\.U\_\{r\}\(0^\{\+\}\)=H\(p\_\{r\}\)\+\\sum\_\{i:p\_\{ri\}\>0\}p\_\{ri\}\\log\\rho\_\{ri\}=H\(Y\_\{r\}\\mid b\_\{Y\_\{r\}\},\\mathcal\{H\}=h\)\.\(34\)Distinct codes giveUr​\(0\+\)=0U\_\{r\}\(0^\{\+\}\)=0\. Duplicate codes can leave positive uncertainty even without encoding noise\. Letℬβ​\(h\)\\mathcal\{B\}\_\{\\beta\}\(h\)denote either budget policy in Appendix[C\.4](https://arxiv.org/html/2610.12904#A3.SS4)\. IfU¯​\(0\+\)\>ℬβ​\(h\)\\overline\{U\}\(0^\{\+\}\)\>\\mathcal\{B\}\_\{\\beta\}\(h\), the budget is infeasible; equality can also leave zero as the only feasible scale\. An infeasible budget retains the original encoding and uses the sampled one\-hot target, keeping the task subject to the same acceptance rules\. The positive\-channel entropy guarantee applies to branches on which calibration returns a feasibleσ\>0\\sigma\>0\.

ForH¯\>0\\overline\{H\}\>0, calibration first checks the zero\-scale limit and any finite cap\. A feasible finite cap is the maximum in Equation \([7](https://arxiv.org/html/2610.12904#S3.E7)\)\. Otherwise, bisection maintains a feasible lower endpoint and an infeasible upper endpoint\. For an uncapped search, we useβ<1\\beta<1, soℬβ​\(h\)<H¯\\mathcal\{B\}\_\{\\beta\}\(h\)<\\overline\{H\}under either policy\. SinceU¯​\(s\)→H¯\\overline\{U\}\(s\)\\to\\overline\{H\}ass→∞s\\to\\infty, repeatedly doubling a positive endpoint initialized from a nonzero code distance eventually brackets the budget threshold\. The implementation uses 24 bisection steps and returns the feasible lower endpoint, preserving the constraint in exact arithmetic\.

Effectively deterministic tables and searches that resolve no positive scale retain the original encoding\. Their supervision follows the branch rule: use terminal probabilities when the final input and acceptance are independent of the remaining query draw, and sampled hard labels otherwise\. This is separate from the explicit one\-hot rule for an infeasible budget\. Withβ=1\\beta=1, the implementation requires a finite cap to define a finite scale choice for every task\.

The distance matrix and entropy bound are evaluated in double precision, with zero\-probability terms handled by continuity\. The distance matrix is computed once, at costO⁡\(K2​d\)O\(K^\{2\}d\), and each bound evaluation costsO⁡\(L​K2\)O\(LK^\{2\}\); rows can be processed in chunks\. Calibration uses1L​∑rUr​\(s\)\\frac\{1\}\{L\}\\sum\_\{r\}U\_\{r\}\(s\), which generally differs from evaluating the bound at the mean row probabilities\. The selected scale is used consistently in both the Gaussian draw and its posterior likelihood\.

### C\.3Average information guarantee and its scope

For any feasible positive scale selected atℋ=h\\mathcal\{H\}=h, averaging Equation \([6](https://arxiv.org/html/2610.12904#S3.E6)\) and applying the feasibility constraint in Equation \([7](https://arxiv.org/html/2610.12904#S3.E7)\) gives

1L​∑r=1Lℰr​\(σ⋆\)≤U¯​\(σ⋆\)≤ℬβ​\(h\)≤β​H¯\.\\frac\{1\}\{L\}\\sum\_\{r=1\}^\{L\}\\mathcal\{E\}\_\{r\}\(\\sigma\_\{\\star\}\)\\leq\\overline\{U\}\(\\sigma\_\{\\star\}\)\\leq\\mathcal\{B\}\_\{\\beta\}\(h\)\\leq\\beta\\overline\{H\}\.\(35\)UsingI⁡\(Yr;Wr∣ℋ=h\)=H⁡\(pr\)−ℰr​\(σ⋆\)I\(Y\_\{r\};W\_\{r\}\\mid\\mathcal\{H\}=h\)=H\(p\_\{r\}\)\-\\mathcal\{E\}\_\{r\}\(\\sigma\_\{\\star\}\)also bounds the retained mutual information:

1L​∑r=1LH⁡\(Yr∣Wr,ℋ=h\)\\displaystyle\\frac\{1\}\{L\}\\sum\_\{r=1\}^\{L\}H\(Y\_\{r\}\\mid W\_\{r\},\\mathcal\{H\}=h\)≤ℬβ​\(h\),\\displaystyle\\leq\\mathcal\{B\}\_\{\\beta\}\(h\),\(36\)1L​∑r=1LI⁡\(Yr;Wr∣ℋ=h\)\\displaystyle\\frac\{1\}\{L\}\\sum\_\{r=1\}^\{L\}I\(Y\_\{r\};W\_\{r\}\\mid\\mathcal\{H\}=h\)≥H¯−ℬβ​\(h\)≥\(1−β\)​H¯\.\\displaystyle\\geq\\overline\{H\}\-\\mathcal\{B\}\_\{\\beta\}\(h\)\\geq\(1\-\\beta\)\\overline\{H\}\.Here the channel usesσ⋆​\(h\)\\sigma\_\{\\star\}\(h\)\.

The averaging is over allLLrows and over the label and encoding draws conditional on the state before task filtering\. Thus Equation \([36](https://arxiv.org/html/2610.12904#A3.E36)\) specifies an all\-row, pre\-selection guarantee for the local encoding channel\. Within this constraint, calibration selects the largest certified scale; the resulting conditional supervision satisfies Equation \([31](https://arxiv.org/html/2610.12904#A2.E31)\)\.

### C\.4Relative and binary\-cap budgets

The two implemented budget policies differ only in the allowed residual entropy:

ℬβ​\(h\)=\{β​H¯,relative,β​min⁡\{H¯,log⁡2\},binary cap\.\\mathcal\{B\}\_\{\\beta\}\(h\)=\\begin\{cases\}\\beta\\overline\{H\},&\\text\{relative\},\\\\ \\beta\\min\\\{\\overline\{H\},\\log 2\\\},&\\text\{binary cap\}\.\\end\{cases\}\(37\)The cap is applied after averaging the row entropies\. For binary tasks,H¯≤log⁡2\\overline\{H\}\\leq\\log 2, so both policies give the same budget and scale choice\. For multiclass tasks withH¯\>log⁡2\\overline\{H\}\>\\log 2, the binary cap limits the absolute residual\-entropy allowance toβ​log⁡2\\beta\\log 2\. Under either policy, the same upper bound, posterior, and root\-finding procedure apply\. Sinceℬβ​\(h\)≤β​H¯\\mathcal\{B\}\_\{\\beta\}\(h\)\\leq\\beta\\overline\{H\}, both satisfy the information guarantee in Equation \([36](https://arxiv.org/html/2610.12904#A3.E36)\)\.

The implementation’starget\_encoding\_noise\_max\_stdvalue of zero denotes the uncapped choiceσmax=∞\\sigma\_\{\\max\}=\\infty\. The solver still determines a task\-dependent positive scale whenever the budget and numerical resolution permit it\. The all\-row policy described here computes one budget from all row probabilities and applies the resulting scale to both context and query encodings\.

## Appendix DExperimental Details

### D\.1Pretraining and checkpoints

All six TFM conditions are pretrained for this study; Hard uses the original GraphSCM generator and hard query labels\. The implementations have 27\.55M parameters for TabICLv2, 8\.20M for TabSwift, and 10\.72M for TabPFN\-2\.5\. The TabPFN\-2\.5 implementation uses 24 layers, a hidden dimension of 192, and three attention heads\. Within each architecture, Hard and GIST use the same effective global batch sizes and learning\-rate schedule\. Training follows the two\-stage protocol in Section[4\.1](https://arxiv.org/html/2610.12904#S4.SS1)\. Stage 2 starts from the corresponding condition’s own Stage 1 model, so the downstream comparisons measure accumulated training effects\. Checkpoint selection is fixed across datasets\.

GIST uses a task\-level entropy fraction of 0\.15, all\-row averaging, the binary\-cap policy of Appendix[C\.4](https://arxiv.org/html/2610.12904#A3.SS4), selective conditional targets, and no external Gaussian\-scale cap\. The model architecture and inference configuration are matched within each Hard/GIST pair\.

### D\.2Benchmark protocols and aggregation

#### TabArena\.

We evaluate 30 binary and eight multiclass datasets over all 594 official outer splits in the archived protocol\. The TFM runs use the default benchmark wrappers, eight\-fold internal validation, and full\-data refitting\. TabICLv2 uses an ensemble size of eight, TabSwift uses 16, and TabPFN\-2\.5 uses the archived wrapper defaults\. These internal procedures are distinct from the official outer test splits used to compute scores\. The five conventional baselines reuse the benchmark’s tuned single\-model results on the same outer splits\.

#### TALENT\.

For each dataset, the original train, validation, and test partitions are combined and repartitioned with stratified five\-fold cross\-validation using a fixed seed\. TFMs use an ensemble size of eight\. Baseline hyperparameters are transferred from available TALENT configurations or selected with at most 30 trials or 20 minutes per dataset and method\. Additional searches for GBDTs, RealMLP, and TabM optimize binary ROC\-AUC or multiclass log loss using the original train/validation split and two additional stratified holdouts from the combined training and validation data\. The selected configuration is reused across all five outer folds, with stopping iterations selected within each outer training fold before refitting\. This transferred\-hyperparameter protocol is not nested cross\-validation\. Log loss is computed from normalized class probabilities in FP64, with probabilities clipped below at10−1510^\{\-15\}\.

The main comparison contains 188 datasets shared by all methods, with 940 observed folds per method\. The extended paired comparisons use each architecture’s complete supported set\.

#### BeyondArena\.

The comparison with external baselines uses the 12 complete datasets shared by all methods: six binary and six multiclass tasks, comprising four temporal and eight grouped datasets\. All 77 outer splits have at most 100K training rows\. We use the official outer splits and default TFM preprocessing wrappers with no inner cross\-validation; the reused tuned baselines retain their benchmark training and validation procedures\. The data revision is2ecfe882ccfb814fc27c4de10a64ceefd5d7655c\.

The broader non\-IID classification collection contains 21 datasets and 92 outer splits\. The complete paired scope covers all 21 datasets for TabICLv2, 18 for TabSwift, and 12 for TabPFN\-2\.5\. The larger\-table runs use the ensemble settings recorded in their evaluation configurations, including ensemble size one for TabSwift above 100K training rows\. Only datasets with every official split completed for both conditions enter a paired comparison\. Incomplete or cancelled evaluations receive no imputed score\. The main\-text OOD win rates in Table[3](https://arxiv.org/html/2610.12904#S4.T3)use these architecture\-specific sets\. Table[4](https://arxiv.org/html/2610.12904#A4.T4)reports their win/loss/tie counts, together with the extended IID ranges\.

Table 4:Stage 2 paired results on each architecture’s complete supported range\. These ranges differ across architectures and are not used for cross\-architecture rankings\. W/L/T denotes wins/losses/ties of GIST against Hard\.BenchmarkArchitectureDatasetsW/L/TWin \(%\)TabArenaTabICLv23821/17/055\.3TabArenaTabSwift3825/13/065\.8TabArenaTabPFN\-2\.53827/11/071\.1TALENTTabICLv2200132/65/366\.0TALENTTabSwift200137/62/168\.5TALENTTabPFN\-2\.5188101/85/253\.7BeyondArenaTabICLv22111/10/052\.4BeyondArenaTabSwift1815/3/083\.3BeyondArenaTabPFN\-2\.5127/5/058\.3
#### Metrics\.

Within a benchmark and training stage, all 11 methods are ranked on the same dataset/split set\. For methodmmon splitssof datasetdd, defineem,d,se\_\{m,d,s\}as1−AUC1\-\\mathrm\{AUC\}for binary tasks or log loss for multiclass tasks\. The split\-level version of TabArena’s Improvability\([Erickson et al\., 2025](https://arxiv.org/html/2610.12904#bib.bib21)\)is

Im,d,s=1−minj⁡ej,d,sem,d,s,Im,d,s=0​if​em,d,s=0\.I\_\{m,d,s\}=1\-\\frac\{\\min\_\{j\}e\_\{j,d,s\}\}\{e\_\{m,d,s\}\},\\qquad I\_\{m,d,s\}=0\\ \\text\{if \}e\_\{m,d,s\}=0\.\(38\)We average split\-level ranks and Improvability within each dataset, then average across datasets with equal weight\. This prevents datasets with more outer splits from receiving greater aggregate weight\. The best\-error reference and the ranks are computed separately for each stage’s method set\. The five conventional baselines retain the same predictions at both stages\.

Paired wins instead compare the mean raw metric within each dataset\. Higher AUC or lower log loss counts as a win; differences within10−1210^\{\-12\}count as ties\. Win rate isW/\(W\+L\+T\)W/\(W\+L\+T\)\. Mean rank, Improvability, and win rate summarize different aspects of performance and need not agree in direction\.

### D\.3Detailed OOD results

Table[6](https://arxiv.org/html/2610.12904#A4.T6)gives the raw metrics and aggregate results on the 12 common BeyondArena datasets used in Table[5](https://arxiv.org/html/2610.12904#A4.T5)\. The two win\-rate columns in Table[5](https://arxiv.org/html/2610.12904#A4.T5)use each architecture’s paired outcomes: Hard wins are GIST losses, while ties remain in both denominators\. At Stage 2 there are no ties on this common set\. GIST lowers mean Improvability for all three architectures; TabSwift also improves mean rank\. TabM has the lowest Improvability among the 11 methods, while the Hard TabPFN\-2\.5 control has the lowest mean rank\.

Table 5:Paired OOD win rates on 12 BeyondArena datasets\. Each column reports its dataset\-level wins against the other condition\. There are no ties\.ArchitectureHardGISTTabICLv250\.0%50\.0%TabSwift25\.0%75\.0%TabPFN\-2\.541\.7%58\.3%Table 6:Stage 2 OOD results on the 12 common BeyondArena datasets \(six binary, six multiclass\)\. AUC is binary ROC\-AUC; LL is multiclass log loss;IIis mean Improvability \(%\); Rank is mean rank among 11 methods\. Scores are averaged within each dataset and then equally across datasets\. Win compares GIST with its same\-architecture Hard control\. Bold marks the best AUC, LL,II, and Rank\.MethodAUC↑\\uparrowLL↓\\downarrowI↓I\\downarrowRank↓\\downarrowWin \(%\)CatBoost0\.78060\.853720\.254\.85—LightGBM0\.78300\.863821\.426\.06—XGBoost0\.77910\.967026\.636\.59—RealMLP0\.79430\.850818\.645\.37—TabM0\.78940\.813316\.754\.87—TabICLv2 \(Hard\)0\.75270\.865527\.457\.21—TabICLv2 \+ GIST0\.75980\.855727\.337\.2650\.0TabSwift \(Hard\)0\.77160\.937627\.127\.81—TabSwift \+ GIST0\.77710\.930425\.057\.1175\.0TabPFN\-2\.5 \(Hard\)0\.80310\.806418\.314\.19—TabPFN\-2\.5 \+ GIST0\.80020\.788017\.954\.6858\.3
### D\.4Gradient diagnostics

The main grid fixes weights from the GIST training trajectory at initialization, Stage 1 1K/10K/20K/30K, and Stage 2 3K/6K\. Each checkpoint is evaluated at total table lengthsL=1024L=1024and81928192, yielding 42 settings\. The corresponding context/query counts are819/205819/205and6553/16396553/1639\. The archived field namedcontextrecordsLL, rather than the number of labeled context rows\. Hard and GIST share the model forward pass, query mask, and equal\-table/equal\-query loss normalization\. Dropout is disabled, and no optimizer step, weight decay update, or gradient scaler is applied\.

For each setting, we drawP=32P=32independent pairs of batches, with 16 tables per batch\. WritingGai,1G\_\{a\}^\{i,1\}andGai,2G\_\{a\}^\{i,2\}for the two independent batch gradients under supervisiona∈\{H,G\}a\\in\\\{\\mathrm\{H\},\\mathrm\{G\}\\\}\(Hard or GIST\), we clip each at global norm one:G~ai,j=Gai,j​min⁡\{1,\(‖Gai,j‖2\+10−6\)−1\}\\widetilde\{G\}\_\{a\}^\{i,j\}=G\_\{a\}^\{i,j\}\\min\\\{1,\(\\\|G\_\{a\}^\{i,j\}\\\|\_\{2\}\+10^\{\-6\}\)^\{\-1\}\\\}\. The primary diagnostic uses

v^a=12​P​∑i=1P‖G~ai,1−G~ai,2‖22,ρ^=1−v^Gv^H\.\\widehat\{v\}\_\{a\}=\\frac\{1\}\{2P\}\\sum\_\{i=1\}^\{P\}\\left\\\|\\widetilde\{G\}\_\{a\}^\{i,1\}\-\\widetilde\{G\}\_\{a\}^\{i,2\}\\right\\\|\_\{2\}^\{2\},\\qquad\\widehat\{\\rho\}=1\-\\frac\{\\widehat\{v\}\_\{\\mathrm\{G\}\}\}\{\\widehat\{v\}\_\{\\mathrm\{H\}\}\}\.\(39\)Eachv^a\\widehat\{v\}\_\{a\}estimates the trace of the clipped batch\-gradient covariance\. We resample complete batch pairs 4,000 times, keeping both supervision rules paired, to obtain pointwise 95% bootstrap intervals forρ^\\widehat\{\\rho\}that quantify batch\-sampling variability at fixed weights\.

#### Task populations\.

The main grid samples tasks with tractable conditional query targets from the calibrated GraphSCM mixture, as determined by converter type, pre\-draw calibration, and target\-to\-feature topology\. This selection does not depend on sampled query labels, loss, or gradients\. These tasks use terminal probabilities or Gaussian\-channel posteriors and are labeled Tractable targets in Table[7](https://arxiv.org/html/2610.12904#A4.T7)\. The table additionally evaluates the full calibrated mixture, channel\-only tasks, and the same task set restricted to 128 supervised queries at Stage 2 6K, yielding 18 control settings\. Full\-mixture query coverage is approximately 26\.2% atL=1024L=1024and 30\.7% atL=8192L=8192; the remainder is largely hard\-label fallback\. The diagnostic uses fixed table lengths and independent shape draws, so its batch distribution differs from the complete pretraining stream\. All Stage 2 6K diagnostics use the same final decay checkpoints as predictive evaluation\.

Table 7:Gradient diagnostics at the final checkpoints after global norm\-one clipping of each whole\-batch gradient\. RB coverage is the fraction of supervised query rows using conditional targets\. Intervals are pointwise 95% batch\-pair bootstrap intervals\.ArchitecturePoolLLReduction \(%\)RB \(%\)TabICLv2Full1,024\+1\.37​\[−0\.44,\+3\.23\]\+1\.37\\;\[\-0\.44,\+3\.23\]26\.2TabICLv2Channel1,024\+2\.07​\[−0\.77,\+5\.06\]\+2\.07\\;\[\-0\.77,\+5\.06\]98\.5TabICLv2Tractable targets1,024\+11\.52​\[\+6\.72,\+15\.93\]\+11\.52\\;\[\+6\.72,\+15\.93\]99\.3TabICLv2Tractable targets,Q=128Q=1281,024\+13\.77​\[\+9\.50,\+18\.32\]\+13\.77\\;\[\+9\.50,\+18\.32\]99\.4TabICLv2Full8,192\+1\.69​\[\+0\.81,\+3\.07\]\+1\.69\\;\[\+0\.81,\+3\.07\]30\.7TabICLv2Channel8,192\+1\.99​\[−0\.81,\+5\.68\]\+1\.99\\;\[\-0\.81,\+5\.68\]98\.3TabICLv2Tractable targets8,192\+6\.80​\[\+3\.16,\+11\.24\]\+6\.80\\;\[\+3\.16,\+11\.24\]99\.2TabICLv2Tractable targets,Q=128Q=1288,192\+18\.64​\[\+13\.77,\+23\.76\]\+18\.64\\;\[\+13\.77,\+23\.76\]99\.2TabSwiftFull1,024\+3\.37​\[\+2\.36,\+4\.61\]\+3\.37\\;\[\+2\.36,\+4\.61\]26\.2TabSwiftChannel1,024\+6\.82​\[\+4\.51,\+9\.32\]\+6\.82\\;\[\+4\.51,\+9\.32\]98\.5TabSwiftTractable targets1,024\+20\.91​\[\+18\.06,\+24\.03\]\+20\.91\\;\[\+18\.06,\+24\.03\]99\.3TabSwiftTractable targets,Q=128Q=1281,024\+22\.71​\[\+19\.08,\+26\.28\]\+22\.71\\;\[\+19\.08,\+26\.28\]99\.4TabSwiftFull8,192−0\.002​\[−0\.99,\+0\.90\]\-0\.002\\;\[\-0\.99,\+0\.90\]30\.7TabSwiftChannel8,192\+6\.36​\[\+2\.08,\+12\.76\]\+6\.36\\;\[\+2\.08,\+12\.76\]98\.3TabSwiftTractable targets8,192\+10\.17​\[\+4\.53,\+17\.15\]\+10\.17\\;\[\+4\.53,\+17\.15\]99\.2TabSwiftTractable targets,Q=128Q=1288,192\+27\.94​\[\+22\.21,\+34\.13\]\+27\.94\\;\[\+22\.21,\+34\.13\]99\.2TabPFN\-2\.5Full1,024\+7\.33​\[\+5\.78,\+8\.97\]\+7\.33\\;\[\+5\.78,\+8\.97\]26\.2TabPFN\-2\.5Channel1,024\+9\.52​\[\+8\.26,\+10\.79\]\+9\.52\\;\[\+8\.26,\+10\.79\]98\.5TabPFN\-2\.5Tractable targets1,024\+26\.04​\[\+23\.88,\+28\.30\]\+26\.04\\;\[\+23\.88,\+28\.30\]99\.3TabPFN\-2\.5Tractable targets,Q=128Q=1281,024\+27\.22​\[\+24\.87,\+29\.71\]\+27\.22\\;\[\+24\.87,\+29\.71\]99\.4TabPFN\-2\.5Full8,192\+4\.13​\[\+1\.76,\+8\.76\]\+4\.13\\;\[\+1\.76,\+8\.76\]30\.7TabPFN\-2\.5Channel8,192\+4\.57​\[\+3\.27,\+5\.94\]\+4\.57\\;\[\+3\.27,\+5\.94\]98\.3TabPFN\-2\.5Tractable targets8,192\+19\.30​\[\+15\.19,\+23\.71\]\+19\.30\\;\[\+15\.19,\+23\.71\]99\.2TabPFN\-2\.5Tractable targets,Q=128Q=1288,192\+34\.41​\[\+29\.31,\+39\.80\]\+34\.41\\;\[\+29\.31,\+39\.80\]99\.2

### D\.5Functional components after clipping

#### Additive decomposition\.

We partition the trainable parameter coordinates after applying one global clipping factor to each whole\-batch gradient\. The estimatev^a,m\\widehat\{v\}\_\{a,m\}is obtained by restricting the squared distances in Equation \([39](https://arxiv.org/html/2610.12904#A4.E39)\) to groupmm\. The squared Euclidean norm and the covariance trace add over disjoint coordinate groups, sov^a=∑mv^a,m\\widehat\{v\}\_\{a\}=\\sum\_\{m\}\\widehat\{v\}\_\{a,m\}without assuming independence between groups\. We report the Hard variance sharesms\_\{m\}, within\-component relative reductionrmr\_\{m\}, and contribution to the whole\-model reductioncmc\_\{m\}:

sm=v^H,mv^H,rm=100​\(1−v^G,mv^H,m\),cm=100​v^H,m−v^G,mv^H\.s\_\{m\}=\\frac\{\\widehat\{v\}\_\{\\mathrm\{H\},m\}\}\{\\widehat\{v\}\_\{\\mathrm\{H\}\}\},\\qquad r\_\{m\}=100\\left\(1\-\\frac\{\\widehat\{v\}\_\{\\mathrm\{G\},m\}\}\{\\widehat\{v\}\_\{\\mathrm\{H\},m\}\}\\right\),\\qquad c\_\{m\}=100\\frac\{\\widehat\{v\}\_\{\\mathrm\{H\},m\}\-\\widehat\{v\}\_\{\\mathrm\{G\},m\}\}\{\\widehat\{v\}\_\{\\mathrm\{H\}\}\}\.\(40\)Herermr\_\{m\}is defined forv^H,m\>0\\widehat\{v\}\_\{\\mathrm\{H\},m\}\>0and expressed as a percentage;cmc\_\{m\}is a percentage\-point contribution to the whole\-model reductionRR\. The contributions satisfy

cm=sm​rm,∑mcm=R=100​\(1−v^Gv^H\)\.c\_\{m\}=s\_\{m\}r\_\{m\},\\qquad\\sum\_\{m\}c\_\{m\}=R=100\\left\(1\-\\frac\{\\widehat\{v\}\_\{\\mathrm\{G\}\}\}\{\\widehat\{v\}\_\{\\mathrm\{H\}\}\}\\right\)\.\(41\)We compute ratios after summing and averaging the unrounded batch\-pair distances\. All groups and both supervision rules use the same 4,000 resamples of complete batch pairs\. Within each resample, group variances add to the full\-model variance, shares sum to one, and contributions sum to the net reduction\. Tables[8](https://arxiv.org/html/2610.12904#A4.T8)and[9](https://arxiv.org/html/2610.12904#A4.T9)give all three statistics at 36K\. At these final checkpoints, the percentage of the net reduction contributed by components outside the prediction head is100​\(R−chead\)/R100\(R\-c\_\{\\mathrm\{head\}\}\)/R\.

A component’s contribution depends on both its Hard variance share and its relative reduction\. For example, at 36K andL=1024L=1024, TFicl has a larger relative reduction than TFcol, 26\.85% versus 9\.05%, yet contributes less to the whole\-model reduction, 2\.21 versus 7\.87 percentage points\. TFcol accounts for 86\.95% of Hard variance, compared with 8\.23% for TFicl\.

#### Functional partitions\.

TabICLv2 separates three backbone components and the prediction head\. TFcol includes the input projection, its label encoder, and all column\-encoder blocks, including their inducing vectors and SSMax networks\. TFrow includes the row\-interaction blocks, row CLS tokens, and output normalization\. TFicl includes the 12 ICL blocks, its label encoder, and output normalization\. The prediction decoder forms a separate group, shown as Pred\. head in Figure[3](https://arxiv.org/html/2610.12904#S4.F3)\.

TabSwift uses row\-wise attention \(Row attn in Figure[3](https://arxiv.org/html/2610.12904#S4.F3)\), MLP, Encoders, the prediction head, and Other\. Its attention operates across row tokens, with each row embedded as one token and additional learned register tokens\. This group includes query, key, value, and output projections together with attention gates; MLP contains the block feed\-forward layers\. Encoders contains feature and label projections, the prediction head contains the decoder, and Other contains normalization parameters and register tokens\. TabPFN\-2\.5 separates within\-row attention, operating across features of one sample, from within\-column attention, operating across samples in one feature column\. Its other groups are MLP, Encoders \(feature, positional, and label encodings\), the prediction head, and Other \(learned thinking rows\)\. Membership follows parameter paths and is fixed across checkpoints and table lengths\.

#### Parameter coverage\.

Every trainable parameter is assigned to exactly one component, with coverage checked across all 42 main diagnostic settings\. These statistics sum gradient variance over scalar coordinates without normalization by parameter count\.

Table 8:Functional decomposition at the final checkpoints forL=1024L=1024\. Entries are estimates with pointwise 95% batch\-pair bootstrap intervals\. Shares and relative reductions are percentages; contributions are percentage points of whole\-model reduction\. Values are rounded only for display\.ComponentHard share \(%\)Relative reduction \(%\)Contribution \(pp\)TabICLv2TFcol86\.95 \[84\.15, 89\.29\]9\.05 \[3\.57, 13\.94\]7\.87 \[3\.07, 12\.10\]TFrow1\.62 \[1\.29, 2\.01\]10\.14 \[\-3\.97, 19\.36\]0\.16 \[\-0\.07, 0\.32\]TFicl8\.23 \[6\.67, 10\.16\]26\.85 \[23\.55, 29\.96\]2\.21 \[1\.72, 2\.85\]Prediction head3\.20 \[2\.62, 3\.94\]39\.90 \[32\.18, 47\.73\]1\.28 \[0\.94, 1\.73\]TabSwiftRow\-wise attention31\.91 \[28\.55, 35\.09\]20\.01 \[17\.53, 22\.73\]6\.38 \[5\.23, 7\.66\]MLP45\.79 \[41\.89, 50\.15\]19\.95 \[16\.52, 23\.60\]9\.13 \[7\.92, 10\.43\]Encoders12\.40 \[10\.90, 14\.10\]15\.56 \[11\.59, 20\.29\]1\.93 \[1\.56, 2\.34\]Prediction head8\.40 \[6\.72, 10\.23\]37\.70 \[29\.43, 46\.28\]3\.17 \[2\.20, 4\.45\]Other1\.51 \[1\.34, 1\.67\]19\.90 \[17\.17, 22\.92\]0\.30 \[0\.24, 0\.37\]TabPFN\-2\.5Within\-row attention18\.39 \[17\.68, 19\.00\]23\.33 \[20\.67, 25\.74\]4\.29 \[3\.74, 4\.80\]Within\-column attention50\.14 \[48\.73, 51\.58\]24\.08 \[22\.24, 25\.96\]12\.07 \[11\.07, 13\.10\]MLP17\.34 \[16\.92, 17\.74\]23\.19 \[21\.29, 25\.08\]4\.02 \[3\.67, 4\.34\]Encoders0\.21 \[0\.14, 0\.31\]24\.77 \[11\.80, 34\.42\]0\.05 \[0\.02, 0\.10\]Prediction head13\.92 \[12\.51, 15\.47\]40\.28 \[33\.67, 46\.98\]5\.61 \[4\.36, 7\.04\]Other0\.00 \[0\.00, 0\.00\]3\.60 \[\-2\.24, 16\.23\]0\.00 \[\-0\.00, 0\.00\]Table 9:Functional decomposition at the final checkpoints forL=8192L=8192\. Entries are estimates with pointwise 95% batch\-pair bootstrap intervals\. Shares and relative reductions are percentages; contributions are percentage points of whole\-model reduction\. Values are rounded only for display\.ComponentHard share \(%\)Relative reduction \(%\)Contribution \(pp\)TabICLv2TFcol93\.89 \[91\.93, 95\.20\]5\.72 \[1\.93, 10\.24\]5\.37 \[1\.82, 9\.54\]TFrow1\.01 \[0\.79, 1\.29\]8\.35 \[5\.78, 11\.23\]0\.08 \[0\.06, 0\.12\]TFicl3\.88 \[3\.02, 5\.17\]19\.73 \[17\.02, 22\.79\]0\.77 \[0\.57, 1\.07\]Prediction head1\.23 \[0\.92, 1\.71\]47\.23 \[40\.06, 54\.10\]0\.58 \[0\.40, 0\.87\]TabSwiftRow\-wise attention19\.55 \[15\.25, 24\.69\]10\.28 \[4\.92, 16\.72\]2\.01 \[1\.04, 3\.36\]MLP61\.87 \[55\.81, 66\.55\]8\.91 \[2\.44, 16\.41\]5\.51 \[1\.46, 10\.42\]Encoders13\.92 \[12\.42, 15\.90\]8\.02 \[2\.40, 14\.93\]1\.12 \[0\.33, 2\.12\]Prediction head3\.76 \[2\.76, 5\.41\]38\.02 \[29\.22, 46\.81\]1\.43 \[0\.88, 2\.34\]Other0\.90 \[0\.71, 1\.12\]11\.59 \[6\.52, 17\.81\]0\.10 \[0\.06, 0\.17\]TabPFN\-2\.5Within\-row attention17\.17 \[15\.44, 18\.97\]17\.23 \[13\.83, 21\.07\]2\.96 \[2\.43, 3\.60\]Within\-column attention56\.67 \[50\.44, 62\.21\]15\.30 \[10\.84, 19\.67\]8\.67 \[6\.06, 11\.57\]MLP14\.02 \[12\.21, 15\.63\]15\.93 \[12\.58, 19\.84\]2\.23 \[1\.78, 2\.87\]Encoders0\.25 \[0\.11, 0\.50\]10\.86 \[6\.37, 15\.70\]0\.03 \[0\.01, 0\.05\]Prediction head11\.88 \[8\.88, 16\.14\]45\.56 \[36\.77, 53\.39\]5\.41 \[3\.72, 7\.69\]Other0\.00 \[0\.00, 0\.00\]7\.24 \[\-6\.28, 9\.31\]0\.00 \[\-0\.00, 0\.00\]

### D\.6Budget\-sensitivity protocol

The TabSwift sensitivity runs use fixed task\-level budgetsβ∈\{0\.05,0\.15,0\.30\}\\beta\\in\\\{0\.05,0\.15,0\.30\\\}throughout both stages, following the training protocol in Section[4\.1](https://arxiv.org/html/2610.12904#S4.SS1)\. Each Stage 2 run starts from its own budget’s Stage 1 checkpoint\. Model and data seeds are fixed across budgets\. All runs use all\-row averaging, the binary cap, selective fallback, and no external noise cap\. Theβ=0\.15\\beta=0\.15results reuse the main run\.

All budgets use the same 594 TabArena outer splits, default preprocessing, internal validation, and ensemble size 16\. Stage 2 TALENT results use the same 188 datasets as Table[1](https://arxiv.org/html/2610.12904#S4.T1), comprising 120 binary and 68 multiclass tasks, with five stratified folds and ensemble size eight\. All 940 folds per condition are observed; checkpoint selection is fixed across datasets\. Binary AUC and multiclass log loss are averaged separately after computing each dataset’s fold mean\.

Table[10](https://arxiv.org/html/2610.12904#A4.T10)reports the full Stage 2 results, including paired win rates\. Runs follow the protocol in Section[4\.1](https://arxiv.org/html/2610.12904#S4.SS1)\.

Table 10:TabSwift\-GIST entropy\-budget sensitivity at Stage 2\. AUC and LL denote mean binary ROC\-AUC and multiclass log loss; each dataset receives equal weight\. Bold marks the best mean for each metric\. Win is the percentage of datasets won against Hard, including ties in the denominator\. Dataset sets match Table[1](https://arxiv.org/html/2610.12904#S4.T1)\.TabArena \(38\)TALENT \(188\)BudgetAUC↑\\uparrowLL↓\\downarrowWin↑\\uparrowAUC↑\\uparrowLL↓\\downarrowWin↑\\uparrowHard0\.84200\.2848—0\.87130\.4267—β=0\.05\\beta=0\.050\.84270\.282468\.40\.87310\.425064\.9β=0\.15\\beta=0\.150\.84250\.281165\.80\.87260\.424367\.0β=0\.30\\beta=0\.300\.84230\.282260\.50\.87270\.424063\.3