Do Tabular Foundation Models Agree with Themselves?

# Do Tabular Foundation Models Agree with Themselves?
Source: [https://arxiv.org/html/2608.06004](https://arxiv.org/html/2608.06004)
###### Abstract

Tabular Foundation Models \(TFMs\) are currently the best approach to tabular prediction problems\. They are constructed as transformers that approximate the Bayesian posterior predictive distribution based on a pre\-training prior\. These univariate predictors can be converted into multivariate ones autoregressively by sampling one target and adding it to the features\.

However, the faithfulness of the resulting joint has not been investigated\. Furthermore, TFMs cannot be evaluated against the posterior itself, at least not on real\-world datasets, because the ground\-truth distribution is unknown\. We therefore propose asking a different question: could a model’s predictions result from*any*joint distribution? To answer this question, we pose two requirements that any such model must satisfy\. The first is marginalization consistency, which demands that marginalized conditionals are equal to directly predicted marginals\. The second is factorization consistency, which demands that different factorization orders result in equal joint distributions\. Every TFM that we evaluate violates both of these requirements for both classification and regression across all datasets\.

## 1Introduction

Tabular Foundation Models \(TFMs\) rest on a claim about Bayesian inference\.Mülleret al\.\([2022](https://arxiv.org/html/2608.06004#bib.bib4)\)showed that a transformer trained to predict held\-out targets on datasets drawn from a prior approximates the Bayesian posterior predictive distribution \(PPD\) under that prior\. The prior\-data fitted network \(PFN\) blueprint built on this claim now underwrites TFMs such as TabPFN\(Hollmannet al\.[2025](https://arxiv.org/html/2608.06004#bib.bib5)\), TabICL\(Quet al\.[2025](https://arxiv.org/html/2608.06004#bib.bib33)\), TabDPT\(Maet al\.[2025](https://arxiv.org/html/2608.06004#bib.bib34)\)and most recently TabFM\(Kong and Das[2026](https://arxiv.org/html/2608.06004#bib.bib31)\), which are the state\-of\-the\-art for tabular prediction tasks\.

While the prediction heads of current TFMs are restricted to univariate distributions, practitioners use them to model multivariate distributions, relying on the claim that they approximate Bayesian inference\. A value of one target can simply be appended as a feature to condition the prediction of another\. Chaining such conditionals turns a univariate predictor into an autoregressive construction of a multivariate joint\(Vetteret al\.[2025](https://arxiv.org/html/2608.06004#bib.bib6)\)\.

![Refer to caption](https://arxiv.org/html/2608.06004v1/x1.png)Figure 1:A marginalization\-consistency violation on a singleWinetest instance from TabPFNv3\. The violation is observable when comparing panels\(a\)and\(b\), which show distributions that should be equal, but are not\.\(a\)represents the distribution of theQuality\-Ratingusing all predictors butColor\. The distribution depicted in\(b\)is computed by mixing theQuality\-Ratingprediction under the assumption that theColoris red\(c\)and white\(d\), weighted by the probability for each color\(e\)\.What the theory promises in the limit, however, says nothing about how closely a trained model actually attains it\. How closely Bayesian inference is actually approximated by the TFMs has not been analyzed\. It is also unclear how it would be measured, given that only a single sample is available per row\. Hence, the PPD under the pretraining prior is unknown, so there is no reference distribution against which a model’s output could be scored\. We therefore follow a different approach\. Rather than asking how far a model’s predictions sit from the Bayesian posterior, we ask whether they could have come from*any*joint distribution at all\. A well\-defined joint distribution constrains its own marginals and conditionals\. They must obey the law of total probability, and the chain rule must factorize it identically in either order\. These constraints can be checked from a model’s own predictions alone, without any ground\-truth labels and without knowing the prior\. Figure[1](https://arxiv.org/html/2608.06004#S1.F1)shows what a failure looks like using a single row from theWinedataset as an example\. We compare two predicted distributions of theQuality\-Ratingfor some bottle of wine\. The first distribution is predicted directly using all available features but ignoring theColor\. The second distribution is computed by mixing the conditionals under the assumptions that the color is either red or white \(which are all possible wine colors in the dataset\)\. The mixture is weighted by the probability for the respective color\. The true joint distribution over all columns in the dataset is unknown, yet we know that both queries should result in the same marginal distribution ofQuality\-Rating\. TabPFNv3, however, computes two very different distributions, indicating that the model does not follow a well\-defined distribution\.

To guide the ongoing research on TFMs towards minimizing or even eliminating such inconsistencies, we formulate two requirements that any model whose predictions are induced by a well\-defined joint must satisfy\. First,*marginalization consistency*, which ties the model’s conditionals back to its directly predicted marginals\. Second,*factorization consistency*, which demands that the two chain\-rule orders build the same joint distribution\. We formally show that these two consistency requirements are closely connected\. Factorization consistency implies marginalization consistency, but not vice versa\. Hence, every observed marginalization failure already certifies that changing the order of factorization will lead to different distributions\.

We also design experiments to evaluate whether and to what extent different TFM classifiers and \(probabilistic\) regressors violate each consistency requirement\. Every model we evaluate violates both conditions, on every dataset, for both classification and regression\. Our results show that the recently released TabFM is the*most consistent*classifier\. However, this model currently does not offer probabilistic regression\. For regressors we do not find a model that is the most consistent across datasets and requirements\. Furthermore, our experiments show that our proposed measures of violation for each requirement strongly correlate with each other\. Our contributions are:

- •We formulate two consistency requirements that a TFM must satisfy for its implied joint to be well\-defined\.
- •From these requirements we derive metrics that quantify the extent of violations which function without knowledge of the ground truth distribution\.
- •Applying the derived metrics we add to the evaluation of TFMs by comparing them in terms of consistency\. Our results show that all current state\-of\-the\-art TFMs violate both requirements on all datasets\.

## 2Tabular Foundation Models

\(a\) Pretraining???synthetic datasets𝒟\(k\)∼\\mathcal\{D\}^\{\(k\)\}\\simpriorTransformerp^​\(y∣𝐱,𝒟\(k\)\)\\hat\{p\}\\bigl\(y\\mid\\mathbf\{x\},\\mathcal\{D\}^\{\(k\)\}\\bigr\)for held\-out targetsupdateθ\\thetato minimize NLL\(b\) Inference?Transformersingleforward pass1234yyp^​\(y∣𝐱,𝒟\)\\hat\{p\}\(y\\mid\\mathbf\{x\},\\mathcal\{D\}\)context𝒟\\mathcal\{D\}features𝐱\\mathbf\{x\}targetyyFigure 2:The prior\-data fitted network blueprint\. \(a\) A transformer is pretrained to predict held\-out targets on synthetic datasets𝒟\(k\)\\mathcal\{D\}^\{\(k\)\}drawn from a prior\. \(b\) With frozen weights, a single forward pass on a real training set𝒟\\mathcal\{D\}and test point𝐱\\mathbf\{x\}returns a univariate predictive density approximating the posterior predictive under that prior\.Tabular foundation models transfer the in\-context learning paradigm to supervised prediction on tables\. The standard blueprint is the prior\-data fitted network \(PFN; Figure[2](https://arxiv.org/html/2608.06004#S2.F2)\)\. A transformer is pretrained on millions of synthetic datasets drawn from a prior over data\-generating processes \(for instance, randomly sampled structural causal models\) by predicting held\-out target values from the remainder of each dataset\. After pretraining, the model solves a new task without any gradient updates\. It consumes a labeled training set𝒟=\{\(𝐱i,yi\)\}i=1n\\mathcal\{D\}=\{\\\{\(\\mathbf\{x\}\_\{i\},y\_\{i\}\)\\\}\}\_\{i=1\}^\{n\}as its context together with a test feature vector𝐱\\mathbf\{x\}, and a single forward pass returns a predictive distributionp^​\(y∣𝐱;𝒟\)\\hat\{p\}\(y\\mid\\mathbf\{x\};\\mathcal\{D\}\)over the target\. The pretraining objective shapes this output into an approximation of the Bayesian posterior predictive distribution under the synthetic prior\(Mülleret al\.[2022](https://arxiv.org/html/2608.06004#bib.bib4)\)\.

#### One Notation for Both Prediction Tasks

TFMs serve two prediction tasks through two predictive heads: a categorical head over a finite set of classes, and a distributional head for probabilistic regression of a continuous value\. The distributional head is realized as a bin\-wise density\(Hollmannet al\.[2025](https://arxiv.org/html/2608.06004#bib.bib5)\)or as a set of predicted quantiles\(Quet al\.[2026](https://arxiv.org/html/2608.06004#bib.bib2)\), depending on the model\. We treat classification and regression in a unified way\. A targetyjy^\{j\}takes values in a target space𝒴j\\mathcal\{Y\}^\{j\}that is either finite \(classification\) or a continuous subset ofℝ\\mathbb\{R\}\(regression\), andp^\\hat\{p\}always denotes the density of the predictive distribution with respect to the base measure of that space: the counting measure in the finite case, wherep^\\hat\{p\}is a probability mass function and∫𝒴j⋯​dyj\\int\_\{\\mathcal\{Y\}^\{j\}\}\\\!\\cdots\\,\\mathrm\{d\}y^\{j\}is a finite sum, and the Lebesgue measure in the continuous case\. We therefore speak of*densities*and*integrals*throughout, and every statement in this section and in Section[3](https://arxiv.org/html/2608.06004#S3)applies verbatim to classification targets, regression targets, and to a pair of targets of mixed type\.

#### Implied Joint Distributions

While the predictive head is univariate, the in\-context interface is agnostic to which columns act as features and which single column acts as the target\. This flexibility provides conditional predictions for free\. To condition the prediction of one target on the value of another, that value is simply appended as an additional feature column \(to the test point𝐱\\mathbf\{x\}and, using the observed values, to every row of the context set𝒟\\mathcal\{D\}\)\(Vetteret al\.[2025](https://arxiv.org/html/2608.06004#bib.bib6)\)\. Chaining such conditionals turns the univariate predictor into a construction for multivariate joints\. For this,𝐲=\(y1,…,yd\)∈𝒴1×⋯×𝒴d\\mathbf\{y\}=\(y^\{1\},\\dots,y^\{d\}\)\\in\\mathcal\{Y\}^\{1\}\\times\\cdots\\times\\mathcal\{Y\}^\{d\}collectsddtargets of interest\. For any permutationπ\\piof\{1,…,d\}\\\{1,\\dots,d\\\}, the chain rule implies the autoregressive factorization

p^π​\(𝐲∣𝐱\)=∏j=1dp^​\(yπ​\(j\)\|𝐲π\(<j\),𝐱\),\\hat\{p\}\_\{\\pi\}\(\\mathbf\{y\}\\mid\\mathbf\{x\}\)\\;=\\;\\prod\_\{j=1\}^\{d\}\\hat\{p\}\\bigl\(y^\{\\pi\(j\)\}\\,\\big\|\\,\\mathbf\{y\}^\{\\pi\(<j\)\},\\mathbf\{x\}\\bigr\),\(1\)where𝐲π\(<j\)≔\(yπ​\(1\),…,yπ​\(j−1\)\)\\mathbf\{y\}^\{\\pi\(<j\)\}\\coloneqq\(y^\{\\pi\(1\)\},\\dots,y^\{\\pi\(j\-1\)\}\)denotes the already\-processed targets\. Each factor is produced by a separate in\-context inference in which these targets are treated as features\. The context set each factor is conditioned on therefore differs from factor to factor\. We suppress it in \([1](https://arxiv.org/html/2608.06004#S2.E1)\) and make it explicit in Section[3](https://arxiv.org/html/2608.06004#S3), where it carries the argument\. Sampling fromp^π\\hat\{p\}\_\{\\pi\}proceeds ancestrally: drawyπ​\(1\)∼p^​\(yπ​\(1\)∣𝐱\)y^\{\\pi\(1\)\}\\sim\\hat\{p\}\(y^\{\\pi\(1\)\}\\mid\\mathbf\{x\}\), append the sample to the feature vector, drawyπ​\(2\)y^\{\\pi\(2\)\}from the resulting conditional, and so on\.

## 3Consistency Properties

We formalize two requirements for TFMs that are necessary to guarantee that the predicted posteriors stem from a well\-defined multivariate distribution\. We begin with the weaker of the two, and then show that one implies the other\.

#### Notation for a pair of targets\.

Both requirements already apply ford=2d=2, so we state them for two targets and writeAAandBBfor the two target columns,a∈𝒴Aa\\in\\mathcal\{Y\}^\{A\}andb∈𝒴Bb\\in\\mathcal\{Y\}^\{B\}for their values\. Superscripts index target columns throughout:𝒟A\\mathcal\{D\}^\{A\}is the context set in which columnAAis the target andBBis one of the features, and𝒟−BA\\mathcal\{D\}^\{A\}\_\{\-B\}is the same context with columnBBremoved altogether, which is what the model sees when it predictsAAwithout knowingBB\. Ford\>2d\>2both conditions apply verbatim to any pair of targets, with the remaining ones held fixed as features\.

### 3\.1Marginalization Consistency

The most basic requirement ties the model’s conditionals back to its direct univariate predictions\. The autoregressive construction samples the intermediate target from the model’s own predictive distribution; taking this detour must not change what the model*believes*about the remaining target\. Therefore, marginalizing the factorization over the intermediate target must recover the marginal that the model predicts directly\. The following definition is in the spirit of the Kolmogorov Extension Theorem for stochastic processes\(Øksendal[2003](https://arxiv.org/html/2608.06004#bib.bib7)\)\.

###### Definition 1\(Marginalization consistency\)\.

A model is*marginalization\-consistent*iff for every𝐱\\mathbf\{x\}and alla∈𝒴Aa\\in\\mathcal\{Y\}^\{A\},

p^​\(a∣𝐱;𝒟−BA\)=∫𝒴Bp^​\(a∣b,𝐱;𝒟A\)​p^​\(b∣𝐱;𝒟−AB\)​db\.\\hat\{p\}\(a\\mid\\mathbf\{x\};\\mathcal\{D\}^\{A\}\_\{\-B\}\)\\;=\\;\\int\_\{\\mathcal\{Y\}^\{B\}\}\\hat\{p\}\(a\\mid b,\\mathbf\{x\};\\mathcal\{D\}^\{A\}\)\\,\\hat\{p\}\(b\\mid\\mathbf\{x\};\\mathcal\{D\}^\{B\}\_\{\-A\}\)\\,\\mathrm\{d\}b\.\(C1\)

A violation means that the model’s marginal estimate ofaais inconsistent with its own conditional predictions with respect tobb\. The two predictions differ in their context set, i\.e\.,bbis dropped on the left\-hand side but present on the right\-hand side\. For any well\-defined joint distribution the identity holds\. Condition \([C1](https://arxiv.org/html/2608.06004#S3.Ex1)\) demands that the model’s predictions obey the law of total probability; in Bayesian terms, it is an instance of the martingale property of coherent predictive systems\(Bertiet al\.[2004](https://arxiv.org/html/2608.06004#bib.bib13); Fonget al\.[2023](https://arxiv.org/html/2608.06004#bib.bib14)\)\.

### 3\.2Factorization Consistency

The second, stronger requirement concerns the constructed joint distribution itself rather than its univariate projections\. The chain rule factorizes a joint density in either order,p​\(a,b∣𝐱\)=p​\(a∣b,𝐱\)​p​\(b∣𝐱\)=p​\(b∣a,𝐱\)​p​\(a∣𝐱\)p\(a,b\\mid\\mathbf\{x\}\)=p\(a\\mid b,\\mathbf\{x\}\)\\,p\(b\\mid\\mathbf\{x\}\)=p\(b\\mid a,\\mathbf\{x\}\)\\,p\(a\\mid\\mathbf\{x\}\)\. If the model’s predictive densities are induced by a common joint, the two autoregressive constructions must therefore agree, i\.e\.,p^\(a,b\)=p^\(b,a\)\\hat\{p\}\_\{\(a,b\)\}=\\hat\{p\}\_\{\(b,a\)\}\.

###### Definition 2\(Factorization consistency\)\.

A model is*factorization\-consistent*iff for every𝐱\\mathbf\{x\}and alla∈𝒴Aa\\in\\mathcal\{Y\}^\{A\},b∈𝒴Bb\\in\\mathcal\{Y\}^\{B\},

p^​\(a∣b,𝐱;𝒟A\)​p^​\(b∣𝐱;𝒟−AB\)\\displaystyle\\hat\{p\}\(a\\mid b,\\mathbf\{x\};\\mathcal\{D\}^\{A\}\)\\,\\hat\{p\}\(b\\mid\\mathbf\{x\};\\mathcal\{D\}^\{B\}\_\{\-A\}\)\(C2\)=p^​\(b∣a,𝐱;𝒟B\)​p^​\(a∣𝐱;𝒟−BA\)\\displaystyle\\qquad=\\;\\hat\{p\}\(b\\mid a,\\mathbf\{x\};\\mathcal\{D\}^\{B\}\)\\,\\hat\{p\}\(a\\mid\\mathbf\{x\};\\mathcal\{D\}^\{A\}\_\{\-B\}\)

If this property is violated, the autoregressive approach depends on the order in which the targets are sampled\. Equation \([C2](https://arxiv.org/html/2608.06004#S3.Ex2)\) is a compatibility condition, which characterizes when a family of conditionals and marginals can arise from one joint distribution\(Besag[1974](https://arxiv.org/html/2608.06004#bib.bib9); Arnold and Press[1989](https://arxiv.org/html/2608.06004#bib.bib10)\)\.

### 3\.3The Properties Are Strictly Ordered

Although Definitions[1](https://arxiv.org/html/2608.06004#Thmdefinition1)and[2](https://arxiv.org/html/2608.06004#Thmdefinition2)arise from different aspects of coherence \(agreement between conditional and marginal predictions on the one hand, agreement between two factorizations on the other\), they are not independent\. Whenever the predicted conditionals are properly normalized densities, factorization consistency already entails marginalization consistency\.

###### Proposition 1\(\([C2](https://arxiv.org/html/2608.06004#S3.Ex2)\) implies \([C1](https://arxiv.org/html/2608.06004#S3.Ex1)\)\)\.

If a model is factorization\-consistent \(Definition[2](https://arxiv.org/html/2608.06004#Thmdefinition2)\), then it is also marginalization\-consistent \(Definition[1](https://arxiv.org/html/2608.06004#Thmdefinition1)\)\.

###### Proof\.

Let𝐱\\mathbf\{x\}be a feature vector anda∈𝒴Aa\\in\\mathcal\{Y\}^\{A\}\. We show that the RHS of \([C1](https://arxiv.org/html/2608.06004#S3.Ex1)\), the marginal density that the autoregressive construction assigns toaa, reduces top^​\(a∣𝐱;𝒟−BA\)\\hat\{p\}\(a\\mid\\mathbf\{x\};\\mathcal\{D\}^\{A\}\_\{\-B\}\)\.

∫𝒴Bp^​\(a∣b,𝐱;𝒟A\)​p^​\(b∣𝐱;𝒟−AB\)​db\\displaystyle\\int\_\{\\mathcal\{Y\}^\{B\}\}\\hat\{p\}\(a\\mid b,\\mathbf\{x\};\\mathcal\{D\}^\{A\}\)\\,\\hat\{p\}\(b\\mid\\mathbf\{x\};\\mathcal\{D\}^\{B\}\_\{\-A\}\)\\,\\mathrm\{d\}b=\([C2](https://arxiv.org/html/2608.06004#S3.Ex2)\)​∫𝒴Bp^​\(b∣a,𝐱;𝒟B\)​p^​\(a∣𝐱;𝒟−BA\)​db\\displaystyle\\quad\\overset\{\\text\{\\eqref\{eq:factorization\-consistency\}\}\}\{=\}\\int\_\{\\mathcal\{Y\}^\{B\}\}\\hat\{p\}\(b\\mid a,\\mathbf\{x\};\\mathcal\{D\}^\{B\}\)\\,\\hat\{p\}\(a\\mid\\mathbf\{x\};\\mathcal\{D\}^\{A\}\_\{\-B\}\)\\,\\mathrm\{d\}b=p^​\(a∣𝐱;𝒟−BA\)​∫𝒴Bp^​\(b∣a,𝐱;𝒟B\)​db⏟=1\\displaystyle\\quad=\\hat\{p\}\(a\\mid\\mathbf\{x\};\\mathcal\{D\}^\{A\}\_\{\-B\}\)\\underbrace\{\\int\_\{\\mathcal\{Y\}^\{B\}\}\\hat\{p\}\(b\\mid a,\\mathbf\{x\};\\mathcal\{D\}^\{B\}\)\\,\\mathrm\{d\}b\}\_\{=\\,1\}=p^​\(a∣𝐱;𝒟−BA\)\\displaystyle\\quad=\\hat\{p\}\(a\\mid\\mathbf\{x\};\\mathcal\{D\}^\{A\}\_\{\-B\}\)The first step applies \([C2](https://arxiv.org/html/2608.06004#S3.Ex2)\) to the integrand, the second pullsp^​\(a∣𝐱;𝒟−BA\)\\hat\{p\}\(a\\mid\\mathbf\{x\};\\mathcal\{D\}^\{A\}\_\{\-B\}\)out of the integral, since it does not depend onbb\. The third step uses thatp^\(⋅∣a,𝐱;𝒟B\)\\hat\{p\}\(\\,\\cdot\\mid a,\\mathbf\{x\};\\mathcal\{D\}^\{B\}\)is a normalized probability density and therefore integrates to one\. Without the normalization, \([C1](https://arxiv.org/html/2608.06004#S3.Ex1)\) holds only up to some constant factor\. ∎

The contrapositive of[Proposition˜1](https://arxiv.org/html/2608.06004#Thmproposition1)yields the pragmatic approach of only testing for \([C1](https://arxiv.org/html/2608.06004#S3.Ex1)\)\. Yet, since our work aims at fostering the understanding of these two properties, we investigate both of them in the following\. Moreover, we want to quantify the extent to which either condition is violated by TFMs\.

###### Proposition 2\(\([C1](https://arxiv.org/html/2608.06004#S3.Ex1)\) does not imply \([C2](https://arxiv.org/html/2608.06004#S3.Ex2)\)\)\.

Marginalization consistency does not imply factorization consistency\.

###### Proof\.

We provide a counterexample in Appendix[A](https://arxiv.org/html/2608.06004#A1)\. ∎

#### Consistency matters\.

The ground\-truth data\-generating process that any TFM aims to capture*is*a single joint distribution, whose marginals and conditionals cohere by construction\. Both \([C1](https://arxiv.org/html/2608.06004#S3.Ex1)\) and \([C2](https://arxiv.org/html/2608.06004#S3.Ex2)\) therefore necessarily hold for the true underlying distribution\. A model that violates either has provably departed from*any*valid joint\. Therefore, such a model cannot be considered as an accurate estimate of the underlying distribution\.

The incoherence also propagates into any downstream task\. Autoregressive data generation\(Vetteret al\.[2025](https://arxiv.org/html/2608.06004#bib.bib6)\)draws records from an arbitrary and order\-dependent joint\. Therefore, one cannot trust these joints for decisions that are based on the dependence between targets\.

## 4Experiments

We want to put the just\-introduced conditions to a test on current TFMs\. The goal is to detect and quantify to which extent each condition is violated\. The proposed method shall establish a new aspect for analyzing and comparing TFMs and their prediction capabilities\. Furthermore, we examine the relationship between the two consistency requirements\.

### 4\.1Setup

#### Models\.

We conduct our experiments on several state\-of\-the\-art TFMs: TabPFNv2/v3\(Hollmannet al\.[2025](https://arxiv.org/html/2608.06004#bib.bib5); Grinsztajnet al\.[2026](https://arxiv.org/html/2608.06004#bib.bib3)\), whose regression head returns a bin\-wise piecewise\-constant density, TabICLv2\(Quet al\.[2026](https://arxiv.org/html/2608.06004#bib.bib2)\), whose head returns predicted quantiles, andTabDPTv1\.2\(Maet al\.[2025](https://arxiv.org/html/2608.06004#bib.bib34); Hosseinzadehet al\.[2026](https://arxiv.org/html/2608.06004#bib.bib37)\), whose head is a softmax over equal\-width bins of the standardized target space\. For classification, we additionally includeTabICLv1\(Quet al\.[2025](https://arxiv.org/html/2608.06004#bib.bib33)\)and TabFM\(Kong and Das[2026](https://arxiv.org/html/2608.06004#bib.bib31)\)\. While the former does not provide a regressor, the latter offers point regression only\.

#### Datasets\.

We employ tabular datasets from OpenML\(Feureret al\.[2021](https://arxiv.org/html/2608.06004#bib.bib32)\)in which two columns form a natural pair of targets that plausibly depend on each other given the remaining features, with all remaining columns serving as the feature vector𝐱\\mathbf\{x\}\. We evaluate on a broad suite spanning both classification and regression; Appendix[B](https://arxiv.org/html/2608.06004#A2)lists every dataset’s target pair, size, feature counts, and OpenML ID, together with the selection criteria and preprocessing applied\.

#### Measuring disagreement\.

Each check compares two predictive distributions over the same target space, and we quantify their discrepancy with the total\-variation distance

TV​\(p,q\)=12​∑c\|pc−qc\|∈\[0,1\],\\mathrm\{TV\}\(p,q\)=\\tfrac\{1\}\{2\}\\sum\_\{c\}\\lvert p\_\{c\}\-q\_\{c\}\\rvert\\in\[0,1\],\(2\)where the sum runs over the classes \(classification\) or over the cells of a discretization of the target space \(regression\)\.TV\\mathrm\{TV\}equals0iff the distributions coincide and11iff they are disjoint, and it can be read as follows: a value ofttmeans there is an event to which the two sides of the check assign probabilities differing bytt\. BecauseTV\\mathrm\{TV\}is bounded and unit\-free, classification and regression results are reported on the same scale\.

The classification results can be considered*exact*\. The target spaces are finite, so both the marginalizing integral and the two chain\-rule joints are finite sums that we evaluate in closed form from the model’s categorical heads\. Any nonzeroTV\\mathrm\{TV\}is a violation by definition\.

For regression the random variables of interest are continuous, and therefore we cannot enumerate the target space\. The distribution of targets modeled by current TFM implementations can be accessed by their quantile function calledF^−1\\hat\{F\}^\{\-1\}\. From it, we derive a bin\-wise uniform approximation of the density\. Consecutive quantile levels define bins of known probability mass, within which the mass is spread uniformly\. Two predictive distributions can then be compared byTV\\mathrm\{TV\}over these bins as in \([2](https://arxiv.org/html/2608.06004#S4.E2)\); see Appendix[C](https://arxiv.org/html/2608.06004#A3)for details\. We run every check under55\-fold cross\-testing\.

All experiments were run on NVIDIA RTX 2080 Ti and A40 GPUs with 11GB and 48GB of memory, respectively\.

### 4\.2Marginalization Consistency

Table 1:Marginalization\-consistency violations, one column per model: total\-variation distanceTV∈\[0,1\]\\mathrm\{TV\}\\in\[0,1\]between the direct marginalp^​\(a∣𝐱;𝒟−BA\)\\hat\{p\}\(a\\mid\\mathbf\{x\};\\mathcal\{D\}^\{A\}\_\{\-B\}\)and the marginalized mixture, averaged over the test set, with the standard deviation over the five cross\-testing folds as a subscript and the worst single test instance in parentheses\. Larger is worse; a consistent model would score0\. Per row,green \(bold\)andbluemark the smallest and second\-smallest mean gap,redthe largest; the parenthesized worst\-instance values are ranked and colored the same way\. TabICLv1 ships no regressor and TabFM’s regression head emits only point predictions; both are evaluated on classification only\. An asterisk \(\*\) marks datasets included in the TabArena benchmark\.Condition \([C1](https://arxiv.org/html/2608.06004#S3.Ex1)\) asks whether marginalizing the autoregressive factorization over the intermediate targetbbrecovers the directly predicted marginal ofaa\. We form the mixture∫𝒴Bp^​\(a∣b,𝐱;𝒟A\)​p^​\(b∣𝐱;𝒟−AB\)​db\\int\_\{\\mathcal\{Y\}^\{B\}\}\\hat\{p\}\(a\\mid b,\\mathbf\{x\};\\mathcal\{D\}^\{A\}\)\\,\\hat\{p\}\(b\\mid\\mathbf\{x\};\\mathcal\{D\}^\{B\}\_\{\-A\}\)\\,\\mathrm\{d\}bfrom the model’s own conditional and marginal heads and compare it againstp^​\(a∣𝐱;𝒟−BA\)\\hat\{p\}\(a\\mid\\mathbf\{x\};\\mathcal\{D\}^\{A\}\_\{\-B\}\)\.

#### Classification\.

For categorical targets, the integral is a finite sum, so we evaluate the conditionalp^​\(a∣b,𝐱;𝒟A\)\\hat\{p\}\(a\\mid b,\\mathbf\{x\};\\mathcal\{D\}^\{A\}\)at every class ofbb, weight each by the model’s predictedp^​\(b∣𝐱;𝒟−AB\)\\hat\{p\}\(b\\mid\\mathbf\{x\};\\mathcal\{D\}^\{B\}\_\{\-A\}\), and compare the resulting mixture to the direct marginal\. Figure[1](https://arxiv.org/html/2608.06004#S1.F1)already illustrated such a violation on a singleWineinstance\.

*Observations\.*Table[1](https://arxiv.org/html/2608.06004#S4.T1)shows that the violations are systematic\. Every model violates the condition on every dataset\. Since the mixture is evaluated in closed form, these gaps cannot be sampling noise\. The computational detour throughbbtruly alters the model’s estimate ofaa, even though the estimate ofbbis obtained from the TFM itself and provides no additional information\. No single model has the smallest violations on every dataset, so there is no uniformly*most consistent*classifier for marginalization: TabFM has the lowest average TV on five of the ten datasets, TabDPT on four, and TabPFNv3 on the remaining one\. TabICLv1 turns out to be the most inconsistent model\.

#### Regression\.

We cannot enumerate𝒴B\\mathcal\{Y\}^\{B\}, so we discretize it deterministically: we replacep^​\(b∣𝐱;𝒟−AB\)\\hat\{p\}\(b\\mid\\mathbf\{x\};\\mathcal\{D\}^\{B\}\_\{\-A\}\)byK=1,000K=1\{,\}000equal\-mass atomsbk=\(F^B\)−1​\(k−1/2K\)b\_\{k\}=\{\(\\hat\{F\}^\{B\}\)\}^\{\-1\}\\big\(\\tfrac\{k\-1/2\}\{K\}\\big\)taken from the quantile head\. We compute a midpoint quadrature of the marginalization integral and average the conditional*CDFs*,F^mix=1K∑kF^\(⋅∣bk,𝐱;𝒟A\)\\hat\{F\}\_\{\\mathrm\{mix\}\}=\\tfrac\{1\}\{K\}\\sum\_\{k\}\\hat\{F\}\(\\,\\cdot\\mid b\_\{k\},\\mathbf\{x\};\\mathcal\{D\}^\{A\}\)\. The averaged CDF is compared with the direct marginal’s CDF viaTV\\mathrm\{TV\}on a sharedS=20S=20\-cell grid\.

The resolutionsK=1,000K=1\{,\}000andS=20S=20are deliberately set to opposite ends\. A largeKKmakes the quadrature of the marginalization integral as precise as needed, so the mixture is not distorted by a coarse approximation ofp^​\(b∣𝐱\)\\hat\{p\}\(b\\mid\\mathbf\{x\}\)\. The comparison grid is coarsened for the opposite reason: becauseTV\\mathrm\{TV\}is non\-increasing under coarsening\(Csiszár and Shields[2004](https://arxiv.org/html/2608.06004#bib.bib35)\), merging cells can only hide discrepancies that fall within a cell but never create them\. AnS=20S=20\-cell grid is coarse enough for the comparison to be insensitive to the quadrature discretization\. The resulting value forTV\\mathrm\{TV\}is therefore a*lower bound*for any evaluation on a finer grid, and in particular for theTV\\mathrm\{TV\}between the underlying continuous distributions\(Nielsen and Sun[2018](https://arxiv.org/html/2608.06004#bib.bib36)\)\.

*Observations\.*In the regression setting there is not a single TFM that scores an average TV close to zero on any dataset\. Again, TabDPT appears to be the model that comes closest to marginalization consistency, as it has the smallest TV on all datasets butInsurance\. There, however, it has the highest TV among tested models\.

### 4\.3Factorization Consistency

TabPFNv2TabICLv1TabDPTTabICLv2TabPFNv3TabFMClassificationAnneal\*0\.0148±\\pm\.0015\(0\.29\)0\.0261±\\pm\.0015\(0\.36\)0\.0123±\\pm\.0020\(0\.27\)0\.0129±\\pm\.0021\(0\.29\)0\.0173±\\pm\.0023\(0\.42\)0\.0042±\\pm\.0014\(0\.19\)Credit\*0\.0470±\\pm\.0037\(0\.14\)0\.0632±\\pm\.0046\(0\.22\)0\.0510±\\pm\.0020\(0\.24\)0\.0517±\\pm\.0048\(0\.26\)0\.0506±\\pm\.0045\(0\.16\)0\.0484±\\pm\.0044\(0\.18\)Phishing\*0\.0522±\\pm\.0030\(0\.43\)0\.0536±\\pm\.0014\(0\.43\)0\.0436±\\pm\.0024\(0\.29\)0\.0448±\\pm\.0031\(0\.36\)0\.0422±\\pm\.0022\(0\.32\)0\.0409±\\pm\.0032\(0\.34\)MIC\*0\.0277±\\pm\.0026\(0\.16\)0\.0297±\\pm\.0018\(0\.24\)0\.0162±\\pm\.0015\(0\.13\)0\.0254±\\pm\.0006\(0\.16\)0\.0209±\\pm\.0017\(0\.15\)0\.0166±\\pm\.0014\(0\.20\)Customer\*0\.0231±\\pm\.0016\(0\.12\)0\.0252±\\pm\.0007\(0\.11\)0\.0180±\\pm\.0032\(0\.09\)0\.0184±\\pm\.0016\(0\.09\)0\.0185±\\pm\.0025\(0\.08\)0\.0144±\\pm\.0019\(0\.10\)Car0\.0893±\\pm\.0085\(0\.45\)0\.0939±\\pm\.0060\(0\.39\)0\.1658±\\pm\.0374\(0\.55\)0\.0644±\\pm\.0058\(0\.50\)0\.0616±\\pm\.0048\(0\.54\)0\.0216±\\pm\.0038\(0\.18\)Marketing\*0\.0434±\\pm\.0038\(0\.20\)0\.0467±\\pm\.0025\(0\.42\)0\.0312±\\pm\.0059\(0\.39\)0\.0344±\\pm\.0040\(0\.39\)0\.0258±\\pm\.0059\(0\.36\)0\.0219±\\pm\.0013\(0\.22\)Wine\*0\.0487±\\pm\.0018\(0\.59\)0\.0350±\\pm\.0007\(0\.48\)0\.0165±\\pm\.0001\(0\.41\)0\.0193±\\pm\.0007\(0\.72\)0\.0214±\\pm\.0006\(0\.66\)0\.0093±\\pm\.0008\(0\.43\)Nursery0\.0745±\\pm\.0053\(0\.45\)0\.0746±\\pm\.0025\(0\.28\)0\.1120±\\pm\.0043\(0\.43\)0\.0452±\\pm\.0019\(0\.44\)0\.0378±\\pm\.0020\(0\.41\)0\.0138±\\pm\.0007\(0\.25\)Diamonds0\.0655±\\pm\.0005\(0\.38\)0\.0916±\\pm\.0006\(0\.46\)0\.0724±\\pm\.0006\(0\.72\)0\.0744±\\pm\.0010\(0\.61\)0\.0632±\\pm\.0010\(0\.54\)0\.0593±\\pm\.0004\(0\.76\)RegressionBoston0\.1335±\\pm\.0099\(0\.36\)—0\.1426±\\pm\.0091\(0\.37\)0\.4762±\\pm\.0257\(0\.90\)0\.1379±\\pm\.0061\(0\.39\)—QSAR\*0\.2068±\\pm\.0166\(0\.62\)—0\.2116±\\pm\.0110\(0\.90\)0\.2181±\\pm\.0058\(0\.74\)0\.2284±\\pm\.0055\(0\.73\)—Stock0\.1307±\\pm\.0053\(0\.46\)—0\.1448±\\pm\.0081\(0\.40\)0\.1626±\\pm\.0054\(0\.38\)0\.1484±\\pm\.0043\(0\.44\)—Concrete0\.2206±\\pm\.0115\(0\.77\)—0\.1815±\\pm\.0085\(0\.74\)0\.2578±\\pm\.0118\(0\.80\)0\.1813±\\pm\.0067\(0\.68\)—Insurance\*0\.1634±\\pm\.0044\(0\.53\)—0\.2576±\\pm\.0115\(0\.73\)0\.2366±\\pm\.0093\(0\.78\)0\.1620±\\pm\.0100\(0\.74\)—Fiat\*0\.1617±\\pm\.0042\(0\.44\)—0\.1973±\\pm\.0040\(0\.87\)0\.1883±\\pm\.0069\(0\.60\)0\.1464±\\pm\.0044\(0\.62\)—Kin8nm0\.2007±\\pm\.0068\(0\.53\)—0\.1845±\\pm\.0027\(0\.53\)0\.2011±\\pm\.0028\(0\.50\)0\.1570±\\pm\.0028\(0\.43\)—Table 2:Factorization\-consistency violations, one column per model: total\-variation distance between the two chain\-rule joints, averaged over the test set, with the standard deviation over the five cross\-testing folds as a subscript and the worst single test instance in parentheses\. Larger is worse; a consistent model would score0\. Per row,green \(bold\)andbluemark the smallest and second\-smallest mean gap,redthe largest; the parenthesized worst\-instance values are ranked and colored the same way\. TabICLv1 ships no regressor and TabFM’s regression head emits only point predictions; both are evaluated on classification only\. An asterisk \(\*\) marks datasets included in the TabArena benchmark\.Condition \([C2](https://arxiv.org/html/2608.06004#S3.Ex2)\) is the stronger requirement\. It asks whether the joint distribution built by the chain rule is independent of the order in which the two targets are sampled\. We construct both joints and measure how far apart they are\.

#### Classification\.

We build each joint exactly, enumerating over all possiblea,ba,bfor each of the two orders \(p^​\(a∣b,𝐱;𝒟A\)\\hat\{p\}\(a\\mid b,\\mathbf\{x\};\\mathcal\{D\}^\{A\}\),p^​\(b∣a,𝐱;𝒟B\)\\hat\{p\}\(b\\mid a,\\mathbf\{x\};\\mathcal\{D\}^\{B\}\)\)\. The resulting joint distributions are therefore tables of size\|𝒴A\|×\|𝒴B\|\|\\mathcal\{Y\}^\{A\}\|\\times\|\\mathcal\{Y\}^\{B\}\|, where each cell represents a probability for the combination of someaaandbb\. The two tables resulting from each order are then compared based on TV\.

*Observations\.*[Table˜2](https://arxiv.org/html/2608.06004#S4.T2)shows that the two orders disagree everywhere, and by a wider margin than the marginalization check\. This translates to the intuition that follows from[Proposition˜1](https://arxiv.org/html/2608.06004#Thmproposition1): factorization consistency is*harder*to achieve than marginalization consistency\. TabFM sticks out as the TFM that comes closest to fulfilling factorization consistency, as it has the lowest average TV on 8 out of 10 datasets and comes in second on the remaining two\. Similar to the marginalization consistency experiment, TabICLv1 is the classifier with the highest violations on average\.

#### Regression\.

The joint cannot be enumerated, so we discretize each order onto the same128×128128\\times 128grid of\(a,b\)\(a,b\)cells whose axes span the range between the outermost quantiles\. For ordera→ba\\to b, thebb\-axis cell masses and each cell’s probability\-mass midpointbl∗b^\{\*\}\_\{l\}come from the quantile head ofp^​\(b∣𝐱;𝒟−AB\)\\hat\{p\}\(b\\mid\\mathbf\{x\};\\mathcal\{D\}^\{B\}\_\{\-A\}\)\. Then, the conditionalp^​\(a∣bl∗,𝐱;𝒟A\)\\hat\{p\}\(a\\mid b^\{\*\}\_\{l\},\\mathbf\{x\};\\mathcal\{D\}^\{A\}\)distributes each cell’s mass over theaa\-axis through its CDF\. This is again evaluated from the quantile head\. The converse orderb→ab\\to ais constructed analogously\.

*Observations\.*The order\-dependence carries over to the continuous setting\. The two chain\-rule orders thus disagree not only on the marginals but on the dependence structure of the constructed joint, so the*joint distribution*produced by the autoregressive workaround depends on the order in which the targets happen to be sampled\.

In contrast to the marginalization consistency experiment, TabDPT belongs to the*more inconsistent*models in this setting\. TabPFNv3 is the regressor that comes closest to factorization consistency among the evaluated TFMs as it has the lowest average TV on the majority of datasets\.

### 4\.4Discussion

![Refer to caption](https://arxiv.org/html/2608.06004v1/x2.png)Figure 3:Factorization\- versus marginalization\-consistency TV, one marker per \(model, dataset\) pair, filled for classification and open for regression; the dashed line marks equality\. The two gaps are strongly associated \(Pearsonr=0\.89r=0\.89, Spearmanρ=0\.90\\rho=0\.90\)\.Every model violates both conditions on every dataset, so the observed inconsistency is a structural property of current TFMs\. TabFM, the newest and top\-ranked model on TabArena\(Ericksonet al\.[2026](https://arxiv.org/html/2608.06004#bib.bib38)\), comes closest to being consistent among all tested classifiers\. Based on that, one could assume that a model’s consistency automatically improves with its predictive performance\. However, relating each model’s dataset\-level accuracy or RMSE to its consistency gap \(Appendix[D](https://arxiv.org/html/2608.06004#A4)\) reveals no strong correlation\. Therefore, consistency is a distinct axis that benchmarks such as TabArena do not capture so far\.

We find a strong association between the two consistency types \(see Figure[3](https://arxiv.org/html/2608.06004#S4.F3)\)\. Additionally, for the classification task we find that the TV measures for the two consistency requirements are higher for datasets with fewer features\. However, we do not observe this for regression \(see Appendix[E](https://arxiv.org/html/2608.06004#A5)\)\.

TabDPT is a revealing case in two aspects\. First, it is trained with a self\-supervised objective that predicts each column from the others\(Maet al\.[2025](https://arxiv.org/html/2608.06004#bib.bib34); Hosseinzadehet al\.[2026](https://arxiv.org/html/2608.06004#bib.bib37)\)\. As the role of target and feature is repeatedly swapped during training, one would intuitively expect it to align the two conditional directions and thus improve factorization consistency\. This expectation is not met: TabDPT is one of the more inconsistent models under factorization, while it is relatively consistent under marginalization\. Second, unlike the other models, TabDPT is trained not on synthetic but on real data, a corpus that includes those standard OpenML tables in our suite that are not drawn from the curated TabArena re\-uploads\. Yet it is not measurably more consistent on datasets seen during training than on the curated ones it has not seen\. TabArena datasets are marked with an∗\*in[Tables˜1](https://arxiv.org/html/2608.06004#S4.T1)and[2](https://arxiv.org/html/2608.06004#S4.T2)\.

## 5Related Work

#### Coherence of predictive systems\.

That a sequence of one\-step\-ahead predictions defines a valid joint is the defining property of a coherent \(martingale\) predictive system\. This is the foundation of the martingale\-posterior view of Bayesian inference\(Bertiet al\.[2004](https://arxiv.org/html/2608.06004#bib.bib13); Fonget al\.[2023](https://arxiv.org/html/2608.06004#bib.bib14)\)\.Falcket al\.\([2024](https://arxiv.org/html/2608.06004#bib.bib19)\)use exactly this martingale property to test whether in\-context learning in large language models \(LLMs\) is Bayesian\. TabMGP builds martingale posteriors on top of TabPFN, implicitly assuming the requisite coherence\(Nget al\.[2025](https://arxiv.org/html/2608.06004#bib.bib23)\)\.

Our marginalization\-consistency condition is the finite, tabular instance of the martingale property these works rely on\. The results of our experiments show that the condition is violated by TFMs\. This disproves the coherence that such constructions already assume\.

#### Incoherence of any\-order models\.

The tension between univariate training and joint deployment is well documented outside the tabular setting\. Order\-agnostic autoregressive models train a single network to model every conditional ordering\(Uriaet al\.[2014](https://arxiv.org/html/2608.06004#bib.bib16); Hoogeboomet al\.[2022](https://arxiv.org/html/2608.06004#bib.bib17)\), but the orderings do not necessarily agree\.Shihet al\.\([2022](https://arxiv.org/html/2608.06004#bib.bib18)\)show these models are internally redundant, defining the same distribution in several inconsistent ways\.

LLMs likewise return probability judgments that violate the basic properties of probability\(Zhu and Griffiths[2024](https://arxiv.org/html/2608.06004#bib.bib27); Luoet al\.[2025](https://arxiv.org/html/2608.06004#bib.bib22)\)\. We bring this lens to tabular foundation models, and unlike these qualitative diagnoses, we derive label\-free metrics that quantify the extent of inconsistency\.

#### Kolmogorov consistency in neural processes\.

Neural processes are meta\-learned predictors that, like TFMs, map a context set to a predictive distribution\. It is known that they generally fail the Kolmogorov consistency conditions that are required of stochastic processes\(Øksendal[2003](https://arxiv.org/html/2608.06004#bib.bib7)\)\.Young \([2026](https://arxiv.org/html/2608.06004#bib.bib24)\)precisely defines this for conditional neural processes\(Garneloet al\.[2018](https://arxiv.org/html/2608.06004#bib.bib26)\), establishing a*conditioning consistency gap*between adding a point to the context and conditioning on it\. Our marginalization\-consistency condition is the tabular counterpart of this gap\. The difference is that TFMs are deployed with the finite, and often small, context sizes at which such gaps are largest, and we measure the violation empirically rather than bounding it asymptotically\.

Yalavarthiet al\.\([2026](https://arxiv.org/html/2608.06004#bib.bib15)\)define marginalization consistency in the context of probabilistic forecasting for irregularly sampled multivariate time series\. They show experimentally that a previous model fails to fulfill that condition, and introduce a consistent alternative\.

#### Compatibility of conditionals\.

Deciding whether a family of conditional and marginal distributions can arise from a common joint is a fundamental problem in statistics\(Besag[1974](https://arxiv.org/html/2608.06004#bib.bib9); Arnold and Press[1989](https://arxiv.org/html/2608.06004#bib.bib10); Arnoldet al\.[2001](https://arxiv.org/html/2608.06004#bib.bib11)\)\. The same problem underlies dependency networks, where independently fitted conditionals are combined without any guarantee of coherence\(Heckermanet al\.[2000](https://arxiv.org/html/2608.06004#bib.bib12)\)\. Copulas offer the complementary, coherent\-by\-construction route to a joint\(Nelsen[2006](https://arxiv.org/html/2608.06004#bib.bib1)\)by separating the marginals and their dependence structure\. Our consistency conditions reflect the compatibility requirements from the literature, and are turned into tests on a TFM’s predictions\.

## 6Conclusion and Future Work

We began from the observation that the Bayesian promise underlying TFMs cannot be measured directly\. Without access to the pretraining prior, there is no reference posterior predictive distribution to score a model against\. We therefore asked the weaker but checkable question of whether a model’s predictions could have been induced by*any*joint distribution at all, and formalized it as two requirements: marginalization and factorization consistency\. Both are testable from a model’s own outputs alone\. Every TFM we evaluate violates both of them, on every dataset and for classification as well as probabilistic regression\. The answer to our question is therefore negative: the univariate predictions of current TFMs do not arise from any joint distribution over the columns of a table, any of which may be the target of a different query\.

#### Mixed categorical and continuous target pairs\.

The conditions of Section[3](https://arxiv.org/html/2608.06004#S3)are stated over a general target space and apply verbatim to a*mixed*pair \(aacategorical,bbcontinuous\), which would exercise the marginalizing mixture across the categorical/continuous boundary\. That check is uninformative for the models we studied, as today’s tabular foundation models use separate weights for classification and regression, so a mixed pair would compare two effectively different models rather than probe the coherence of one\. It becomes a useful experiment once unified architectures emit classification and regression from shared weights\.

#### Toward consistency\-aware tabular foundation models\.

The conditions we formalize are, at present, purely diagnostic\. No current model is trained or constrained to satisfy them\. We see two complementary lines of work\. The first is*evaluation*: marginalization consistency is cheap to measure, since it compares only univariate heads, and by our contrapositive it already certifies factorization violations\. We therefore argue that it belongs in the standard evaluation suite, alongside accuracy and calibration, for any tabular foundation model that is used, or might be used, as a joint density estimator\.

The second line is*model design*, and centers on how to build in\-context predictors whose univariate predictions provably cohere\. Possible directions include training objectives that penalize marginalization or factorization gaps, architectures that share a common latent joint from which the univariate heads are derived by construction, and post\-hoc reconciliation that projects a set of incompatible conditionals onto the nearest compatible family\. Whether consistency can be achieved without sacrificing expressivity is the central open question\.

#### Native joint prediction\.

Future generations of TFMs may be able to predict multivariate densities or copulas directly\. These models may need to be consistent in a new way\. When queried one target at a time, the model still induces a joint distribution through factorization\. This joint distribution may need to align with the distribution predicted directly by the multivariate head\. Our two conditions then extend from a pair of targets to arbitrary subsets of them\. Regardless of whether columns are part of the features or targets, the induced joint distribution should be consistent\.

## References

- B\. C\. Arnold, E\. Castillo, and J\. M\. Sarabia \(2001\)Conditionally specified distributions: an introduction\.Statistical Science16\(3\),pp\. 249–274\.Cited by:[§5](https://arxiv.org/html/2608.06004#S5.SS0.SSS0.Px4.p1.1)\.
- B\. C\. Arnold and S\. J\. Press \(1989\)Compatible conditional distributions\.Journal of the American Statistical Association84\(405\),pp\. 152–156\.Cited by:[§3\.2](https://arxiv.org/html/2608.06004#S3.SS2.p2.1),[§5](https://arxiv.org/html/2608.06004#S5.SS0.SSS0.Px4.p1.1)\.
- P\. Berti, L\. Pratelli, and P\. Rigo \(2004\)Limit theorems for a class of identically distributed random variables\.The Annals of Probability32\(3\)\.External Links:ISSN 0091\-1798,[Document](https://dx.doi.org/10.1214/009117904000000676)Cited by:[§3\.1](https://arxiv.org/html/2608.06004#S3.SS1.p2.3),[§5](https://arxiv.org/html/2608.06004#S5.SS0.SSS0.Px1.p1.1)\.
- J\. Besag \(1974\)Spatial interaction and the statistical analysis of lattice systems\.Journal of the Royal Statistical Society: Series B \(Methodological\)36\(2\),pp\. 192–236\.Cited by:[§3\.2](https://arxiv.org/html/2608.06004#S3.SS2.p2.1),[§5](https://arxiv.org/html/2608.06004#S5.SS0.SSS0.Px4.p1.1)\.
- I\. Csiszár and P\. C\. Shields \(2004\)Information theory and statistics: A tutorial\.Foundations and Trends in Communications and Information Theory1\(4\),pp\. 417–528\.External Links:[Document](https://dx.doi.org/10.1561/0100000004)Cited by:[§4\.2](https://arxiv.org/html/2608.06004#S4.SS2.SSS0.Px2.p2.8)\.
- N\. Erickson, L\. Purucker, A\. Tschalzev, D\. Holzmüller, P\. Desai, D\. Salinas, and F\. Hutter \(2026\)TabArena: A Living Benchmark for Machine Learning on Tabular Data\.Advances in Neural Information Processing Systems38\.Cited by:[§4\.4](https://arxiv.org/html/2608.06004#S4.SS4.p1.1)\.
- F\. Falck, Z\. Wang, and C\. Holmes \(2024\)Is in\-context learning in large language models Bayesian? A martingale perspective\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235\.Cited by:[§5](https://arxiv.org/html/2608.06004#S5.SS0.SSS0.Px1.p1.1)\.
- M\. Feurer, J\. N\. van Rijn, A\. Kadra, P\. Gijsbers, N\. Mallik, S\. Ravi, A\. Müller, J\. Vanschoren, and F\. Hutter \(2021\)OpenML\-Python: an extensible Python API for OpenML\.Journal of Machine Learning Research22\(100\),pp\. 1–5\.External Links:ISSN 1533\-7928Cited by:[Appendix B](https://arxiv.org/html/2608.06004#A2.p1.2),[§4\.1](https://arxiv.org/html/2608.06004#S4.SS1.SSS0.Px2.p1.1)\.
- E\. Fong, C\. Holmes, and S\. G\. Walker \(2023\)Martingale posterior distributions\.Journal of the Royal Statistical Society Series B: Statistical Methodology85\(5\),pp\. 1357–1391\.Cited by:[§3\.1](https://arxiv.org/html/2608.06004#S3.SS1.p2.3),[§5](https://arxiv.org/html/2608.06004#S5.SS0.SSS0.Px1.p1.1)\.
- M\. Garnelo, D\. Rosenbaum, C\. Maddison, T\. Ramalho, D\. Saxton, M\. Shanahan, Y\. W\. Teh, D\. Rezende, and S\. M\. A\. Eslami \(2018\)Conditional Neural Processes\.InProceedings of the 35th International Conference on Machine Learning,pp\. 1704–1713\.External Links:ISSN 2640\-3498Cited by:[§5](https://arxiv.org/html/2608.06004#S5.SS0.SSS0.Px3.p1.1)\.
- L\. Grinsztajn, K\. Flöge, O\. Key, F\. Birkel, P\. Jund, B\. Roof, M\. Manium, S\. B\. Hoo, M\. Bühler, A\. Garg, D\. Safaric, J\. Robertson, B\. Jäger, S\. Alessi, A\. Hayler, V\. Moroshan, L\. Purucker, P\. Singer, A\. Arazi, J\. Siems, J\. H\. Metzen, G\. Grab, N\. Erickson, S\. Guo, E\. Kalfon, S\. Bing, D\. Salinas, C\. Cornu, L\. C\. Wehrhahn, D\. Kriuchkova, K\. Kaya, L\. Sidhoum, M\. Salmon, J\. Chen, M\. Hulsebos, Y\. LeCun, S\. Müller, B\. Schölkopf, S\. Gambhir, N\. Hollmann, and F\. Hutter \(2026\)TabPFN\-3: Technical Report\.arXiv\.External Links:2605\.13986,[Document](https://dx.doi.org/10.48550/arXiv.2605.13986)Cited by:[§4\.1](https://arxiv.org/html/2608.06004#S4.SS1.SSS0.Px1.p1.1)\.
- D\. Heckerman, D\. M\. Chickering, C\. Meek, R\. Rounthwaite, and C\. Kadie \(2000\)Dependency networks for inference, collaborative filtering, and data visualization\.Journal of Machine Learning Research1,pp\. 49–75\.Cited by:[§5](https://arxiv.org/html/2608.06004#S5.SS0.SSS0.Px4.p1.1)\.
- N\. Hollmann, S\. Müller, L\. Purucker, A\. Krishnakumar, M\. Körfer, S\. B\. Hoo, R\. T\. Schirrmeister, and F\. Hutter \(2025\)Accurate predictions on small data with a tabular foundation model\.Nature637\(8045\),pp\. 319–326\.External Links:ISSN 1476\-4687,[Document](https://dx.doi.org/10.1038/s41586-024-08328-6)Cited by:[§1](https://arxiv.org/html/2608.06004#S1.p1.1),[§2](https://arxiv.org/html/2608.06004#S2.SS0.SSS0.Px1.p1.6),[§4\.1](https://arxiv.org/html/2608.06004#S4.SS1.SSS0.Px1.p1.1)\.
- E\. Hoogeboom, A\. A\. Gritsenko, J\. Bastings, B\. Poole, R\. van den Berg, and T\. Salimans \(2022\)Autoregressive Diffusion Models\.InInternational Conference on Learning Representations,Cited by:[§5](https://arxiv.org/html/2608.06004#S5.SS0.SSS0.Px2.p1.1)\.
- R\. Hosseinzadeh, A\. Labach, Z\. Xue, S\. Han, V\. Thomas, and A\. L\. Caterini \(2026\)TabDPT\-Turbo: Efficient In\-Context Learning for Tabular Prediction\.In2nd ICML Workshop on Foundation Models for Structured Data,Cited by:[§4\.1](https://arxiv.org/html/2608.06004#S4.SS1.SSS0.Px1.p1.1),[§4\.4](https://arxiv.org/html/2608.06004#S4.SS4.p3.1)\.
- W\. Kong and A\. Das \(2026\)Introducing TabFM: A zero\-shot foundation model for tabular data\.Note:https://research\.google/blog/introducing\-tabfm\-a\-zero\-shot\-foundation\-model\-for\-tabular\-data/Accessed: 2026\-07\-20Cited by:[§1](https://arxiv.org/html/2608.06004#S1.p1.1),[§4\.1](https://arxiv.org/html/2608.06004#S4.SS1.SSS0.Px1.p1.1)\.
- X\. Luo, X\. Xu, M\. Ramscar, and B\. C\. Love \(2025\)Probability consistency in large language models: theoretical foundations meet empirical discrepancies\.arXiv\.External Links:2505\.08739,[Document](https://dx.doi.org/10.48550/arXiv.2505.08739)Cited by:[§5](https://arxiv.org/html/2608.06004#S5.SS0.SSS0.Px2.p2.1)\.
- J\. Ma, V\. Thomas, R\. Hosseinzadeh, A\. Labach, J\. Cresswell, K\. Golestan, G\. Yu, A\. L\. Caterini, and M\. Volkovs \(2025\)TabDPT: Scaling Tabular Foundation Models on Real Data\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 172692–172722\.Cited by:[Appendix B](https://arxiv.org/html/2608.06004#A2.SS0.SSS0.Px1.p1.7),[§1](https://arxiv.org/html/2608.06004#S1.p1.1),[§4\.1](https://arxiv.org/html/2608.06004#S4.SS1.SSS0.Px1.p1.1),[§4\.4](https://arxiv.org/html/2608.06004#S4.SS4.p3.1)\.
- S\. Müller, N\. Hollmann, S\. Pineda\-Arango, J\. Grabocka, and F\. Hutter \(2022\)Transformers can do bayesian inference\.InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\-29, 2022,External Links:[Link](https://openreview.net/forum?id=KSugKcbNf9)Cited by:[§1](https://arxiv.org/html/2608.06004#S1.p1.1),[§2](https://arxiv.org/html/2608.06004#S2.p1.3)\.
- R\. B\. Nelsen \(2006\)An introduction to copulas\.2 edition,Springer Series in Statistics,Springer,New York\.External Links:[Document](https://dx.doi.org/10.1007/0-387-28678-0),ISBN 978\-0\-387\-28659\-4Cited by:[§5](https://arxiv.org/html/2608.06004#S5.SS0.SSS0.Px4.p1.1)\.
- K\. Ng, E\. Fong, D\. T\. Frazier, J\. Knoblauch, and S\. Wei \(2025\)TabMGP: martingale posterior with TabPFN\.arXiv\.External Links:2510\.25154,[Document](https://dx.doi.org/10.48550/arXiv.2510.25154)Cited by:[§5](https://arxiv.org/html/2608.06004#S5.SS0.SSS0.Px1.p1.1)\.
- F\. Nielsen and K\. Sun \(2018\)Guaranteed deterministic bounds on the Total Variation distance between univariate mixtures\.In2018 IEEE 28th International Workshop on Machine Learning for Signal Processing \(MLSP\),pp\. 1–6\.External Links:ISSN 1551\-2541,[Document](https://dx.doi.org/10.1109/MLSP.2018.8517093)Cited by:[§4\.2](https://arxiv.org/html/2608.06004#S4.SS2.SSS0.Px2.p2.8)\.
- B\. Øksendal \(2003\)Stochastic Differential Equations\.Universitext,Springer,Berlin, Heidelberg\.External Links:[Document](https://dx.doi.org/10.1007/978-3-642-14394-6),ISBN 978\-3\-540\-04758\-2 978\-3\-642\-14394\-6Cited by:[§3\.1](https://arxiv.org/html/2608.06004#S3.SS1.p1.1),[§5](https://arxiv.org/html/2608.06004#S5.SS0.SSS0.Px3.p1.1)\.
- J\. Qu, D\. Holzmüller, G\. Varoquaux, and M\. L\. Morvan \(2025\)TabICL: A Tabular Foundation Model for In\-Context Learning on Large Data\.InProceedings of the 42nd International Conference on Machine Learning,pp\. 50817–50847\.External Links:ISSN 2640\-3498Cited by:[§1](https://arxiv.org/html/2608.06004#S1.p1.1),[§4\.1](https://arxiv.org/html/2608.06004#S4.SS1.SSS0.Px1.p1.1)\.
- J\. Qu, D\. Holzmüller, G\. Varoquaux, and M\. L\. Morvan \(2026\)TabICLv2: A better, faster, scalable, and open tabular foundation model\.arXiv\.External Links:2602\.11139,[Document](https://dx.doi.org/10.48550/arXiv.2602.11139)Cited by:[§2](https://arxiv.org/html/2608.06004#S2.SS0.SSS0.Px1.p1.6),[§4\.1](https://arxiv.org/html/2608.06004#S4.SS1.SSS0.Px1.p1.1)\.
- A\. Shih, D\. Sadigh, and S\. Ermon \(2022\)Training and Inference on Any\-Order Autoregressive Models the Right Way\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 2762–2775\.Cited by:[§5](https://arxiv.org/html/2608.06004#S5.SS0.SSS0.Px2.p1.1)\.
- B\. Uria, I\. Murray, and H\. Larochelle \(2014\)A Deep and Tractable Density Estimator\.InProceedings of the 31st International Conference on Machine Learning,pp\. 467–475\.External Links:ISSN 1938\-7228Cited by:[§5](https://arxiv.org/html/2608.06004#S5.SS0.SSS0.Px2.p1.1)\.
- J\. Vetter, M\. Gloeckler, D\. Gedon, and J\. H\. Macke \(2025\)Effortless, Simulation\-Efficient Bayesian Inference using Tabular Foundation Models\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 82685–82724\.Cited by:[§1](https://arxiv.org/html/2608.06004#S1.p2.1),[§2](https://arxiv.org/html/2608.06004#S2.SS0.SSS0.Px2.p1.6),[§3\.3](https://arxiv.org/html/2608.06004#S3.SS3.SSS0.Px1.p2.1)\.
- V\. K\. Yalavarthi, R\. Scholz, C\. Klötergens, K\. Madhusudhanan, S\. Born, and L\. Schmidt\-Thieme \(2026\)Reliable Probabilistic Forecasting of Irregular Time Series through Marginalization\-Consistent Flows\.International Conference on Learning Representations2026,pp\. 85403–85427\.Cited by:[§5](https://arxiv.org/html/2608.06004#S5.SS0.SSS0.Px3.p2.1)\.
- R\. Young \(2026\)On the Conditioning Consistency Gap in Conditional Neural Processes\.Transactions on Machine Learning Research\.External Links:ISSN 2835\-8856Cited by:[§5](https://arxiv.org/html/2608.06004#S5.SS0.SSS0.Px3.p1.1)\.
- J\. Zhu and T\. Griffiths \(2024\)Incoherent probability judgments in large language models\.InProceedings of the 46th Annual Meeting of the Cognitive Science Society, CogSci 2024, Rotterdam, The Netherlands, July 24\-27, 2024,L\. K\. Samuelson, S\. Frank, M\. Toneva, A\. Mackey, and E\. Hazeltine \(Eds\.\),External Links:[Link](https://escholarship.org/uc/item/2r68p98c)Cited by:[§5](https://arxiv.org/html/2608.06004#S5.SS0.SSS0.Px2.p2.1)\.

## Appendix ADeferred Proof

We give the construction behind Proposition[2](https://arxiv.org/html/2608.06004#Thmproposition2), which separates the two consistency properties by exhibiting predictive densities that satisfy marginalization consistency in both targets yet violate factorization consistency\.

###### Proposition 2\(Marginalization consistency does not imply factorization consistency, restated\)\.

There exist predictive densities that satisfy marginalization consistency in both targets yet violate factorization consistency\.

###### Construction\.

Fix𝐱\\mathbf\{x\}and take both targets binary,a,b∈\{0,1\}a,b\\in\\\{0,1\\\}\. Let the marginals be uniform,p^​\(a∣𝐱\)=p^​\(b∣𝐱\)=\(12,12\)\\hat\{p\}\(a\\mid\\mathbf\{x\}\)=\\hat\{p\}\(b\\mid\\mathbf\{x\}\)=\(\\tfrac\{1\}\{2\},\\tfrac\{1\}\{2\}\), and set the conditionals \(rows indexed by the free target, columns by the conditioning target\)

p^​\(a∣b,𝐱\)=\(34141434\),p^​\(b∣a,𝐱\)=\(12121212\),\\hat\{p\}\(a\\mid b,\\mathbf\{x\}\)=\\begin\{pmatrix\}\\tfrac\{3\}\{4\}&\\tfrac\{1\}\{4\}\\\\ \\tfrac\{1\}\{4\}&\\tfrac\{3\}\{4\}\\end\{pmatrix\},\\qquad\\hat\{p\}\(b\\mid a,\\mathbf\{x\}\)=\\begin\{pmatrix\}\\tfrac\{1\}\{2\}&\\tfrac\{1\}\{2\}\\\\ \\tfrac\{1\}\{2\}&\\tfrac\{1\}\{2\}\\end\{pmatrix\},so thatbbis independent ofaawhileaadepends onbb\. Both conditionals are properly normalized\. Marginalizingp^​\(a∣b,𝐱\)\\hat\{p\}\(a\\mid b,\\mathbf\{x\}\)against the uniformp^​\(b∣𝐱\)\\hat\{p\}\(b\\mid\\mathbf\{x\}\)gives12​\(34,14\)\+12​\(14,34\)=\(12,12\)=p^​\(a∣𝐱\)\\tfrac\{1\}\{2\}\(\\tfrac\{3\}\{4\},\\tfrac\{1\}\{4\}\)\+\\tfrac\{1\}\{2\}\(\\tfrac\{1\}\{4\},\\tfrac\{3\}\{4\}\)=\(\\tfrac\{1\}\{2\},\\tfrac\{1\}\{2\}\)=\\hat\{p\}\(a\\mid\\mathbf\{x\}\), and marginalizingp^​\(b∣a,𝐱\)\\hat\{p\}\(b\\mid a,\\mathbf\{x\}\)againstp^​\(a∣𝐱\)\\hat\{p\}\(a\\mid\\mathbf\{x\}\)trivially returns\(12,12\)=p^​\(b∣𝐱\)\(\\tfrac\{1\}\{2\},\\tfrac\{1\}\{2\}\)=\\hat\{p\}\(b\\mid\\mathbf\{x\}\); marginalization consistency holds in both targets\. But at\(a,b\)=\(0,0\)\(a,b\)=\(0,0\),

p^\(a=0∣b=0,𝐱\)p^\(b=0∣𝐱\)\\displaystyle\\hat\{p\}\(a\{=\}0\\mid b\{=\}0,\\mathbf\{x\}\)\\,\\hat\{p\}\(b\{=\}0\\mid\\mathbf\{x\}\)=34⋅12=38,\\displaystyle=\\tfrac\{3\}\{4\}\\cdot\\tfrac\{1\}\{2\}=\\tfrac\{3\}\{8\},p^\(b=0∣a=0,𝐱\)p^\(a=0∣𝐱\)\\displaystyle\\hat\{p\}\(b\{=\}0\\mid a\{=\}0,\\mathbf\{x\}\)\\,\\hat\{p\}\(a\{=\}0\\mid\\mathbf\{x\}\)=12⋅12=14,\\displaystyle=\\tfrac\{1\}\{2\}\\cdot\\tfrac\{1\}\{2\}=\\tfrac\{1\}\{4\},so the two sides of \([C2](https://arxiv.org/html/2608.06004#S3.Ex2)\) differ and factorization consistency fails\. The two orders build joints with identical uniform marginals but different dependence: correlated one way, independent the other\. Binary targets serve only to keep the arithmetic explicit; the same construction carries over to continuous target spaces, with the two\-point densities replaced by mixtures of two disjointly supported components\. ∎

## Appendix BDataset Details

All datasets are public tables from OpenML\(Feureret al\.[2021](https://arxiv.org/html/2608.06004#bib.bib32)\), fetched throughfetch\_openmlwith either the version or the numeric dataset ID pinned so that the exact table can be reproduced\. Table[3](https://arxiv.org/html/2608.06004#A2.T3)lists, for each dataset, its size, the split of its feature columns into numerical and categorical, the two targetsaaandbbused in the consistency checks, and its OpenML dataset ID\.

Preprocessing is identical for every dataset and deliberately minimal: rows with missing entries are dropped, the two target columns are removed from the feature set, and every remaining column, numerical or categorical, is passed to the model as a feature𝐱\\mathbf\{x\}\. Apart from the categorical encoding described below, which one of the model APIs requires, no scaling, imputation or feature selection is applied beyond what each model performs internally, so the numbers in Table[3](https://arxiv.org/html/2608.06004#A2.T3)are the tables the models actually see\. Two tables need an exception\.MIC’s112112columns are missing so pervasively that no row is complete and dropping incomplete rows alone would empty the table; there we first drop every column missing more than2%2\\%of its entries \(mostly laboratory values and pre\-admission blood\-pressure readings, which are missing by design rather than at random\), and then drop the remaining incomplete rows, which leaves5353columns and1,5961\{,\}596of the1,6991\{,\}699rows\.Marketingstores the customer enrollment date as free text, which no model in our comparison reads as a date; we parse it to the number of days since the earliest enrollment, turning663663opaque string levels into one ordered numerical feature\. The train/test protocol is the one described in Section[4](https://arxiv.org/html/2608.06004#S4)\.

†Winemerges the red\-wine table \(ID 40691, 1,599 rows\) with the white\-wine table \(ID 40498, 4,898 rows\); the color of the source table is the second target\. ‡MICis the one table in which no row is complete, so columns missing more than2%2\\%of their entries are dropped before incomplete rows are;5353of its112112columns and1,5961\{,\}596of its1,6991\{,\}699rows survive\. §Marketingstores the customer enrollment date as free text; it is counted as a numerical feature because it is parsed to days since the earliest enrollment\.DatasetRowsFeat\.num\./cat\.Targetaa\(main\)Targetbb\(second\)OpenML IDClass\.Anneal898376 / 31classes\(anneal grade\), 5 cl\.steel, 8 cl\.46906Credit1,000197 / 12good\_or\_bad\_customer, 2 cl\.checking\_status, 4 cl\.46918Phishing1,35380 / 8WebsiteType, 3 cl\.SFH, 3 cl\.46963MIC1,596512 / 49LET\_IS\(outcome\), 8 cl\.ASP\_S\_n, 2 cl\.46980‡Customer1,723125 / 7bad\_client\_target, 2 cl\.education, 6 cl\.46938Car1,72850 / 5class\(acceptability\), 4 cl\.safety, 3 cl\.40975Marketing2,2162417 / 7Response\(uptake\), 2 cl\.Marital\_Status, 8 cl\.46940§Wine6,4971111 / 0class\(quality\), 8 cl\.color, 2 cl\.40691†Nursery12,96070 / 7class\(recommendation\), 5 cl\.health, 3 cl\.26Diamonds53,94087 / 1cut, 5 cl\.color, 7 cl\.42225Regr\.Boston5061210 / 2MEDVRM,ρ=0\.70\\rho=0\.70531QSAR90755 / 0LC50MLOGP,ρ=0\.65\\rho=0\.6546954Stock95088 / 0company10company1,ρ=0\.71\\rho=0\.71223Concrete1,03077 / 0strengthwater,ρ=−0\.29\\rho=\-0\.2944959Insurance1,33852 / 3chargesbmi,ρ=0\.20\\rho=0\.2046931Fiat1,53865 / 1pricekm,ρ=−0\.86\\rho=\-0\.8646907Kin8nm8,19277 / 0ytheta1,ρ=−0\.14\\rho=\-0\.14189

Table 3:The OpenML datasets used in the consistency checks\.*Rows*is the number of rows after dropping incomplete records,*Feat\.*the number of feature columns \(all columns except the two targets\), split into numerical and categorical\. Targetaais the main target whose predictive distribution the checks compare; targetbbis the second target that is marginalized over, conditioned on, or sampled first\. For classification targets we give the number of classes, for regression targets the Pearson correlationρ\\rhobetween the two targets\.#### Categorical features\.

The models differ in the input format their public interfaces accept, so a categorical column does not reach all of them by the same route\. TabPFN and TabICL take a data frame and infer which columns are categorical from the column types, so we hand them the table unchanged and leave the encoding to whatever each model does internally\. TabDPT accepts only a numerical matrix \(itsfitrejects anything else, and it applies mean imputation and standardization to every column it is given\), and it offers no way to declare a column categorical, so for TabDPT we integer\-encode the categorical columns ourselves, as the TabDPT evaluation does\(Maet al\.[2025](https://arxiv.org/html/2608.06004#bib.bib34)\)\. The level order is fixed on the training split and reused on the test split, so both encode identically; levels unseen in training map to−1\-1\. Columns whose levels are numbers stored as strings \(Boston’sRADandCHASare nominal in the OpenML table, but their levels are the integers11–88,2424and0/11\) are read as numbers, so their order and spacing survive; only levels that are genuinely non\-numeric, such asFiat’s trim levelpop/lounge/sport, receive codes\. Two remarks\. First, integer encoding imposes an arbitrary order on levels that have none, which costs TabDPT some accuracy on the affected tables \(the ones with a nonzero categorical count in Table[3](https://arxiv.org/html/2608.06004#A2.T3)\), but the same encoding is used in every term of both consistency checks, so it cannot by itself open a gap between two factorization orders or between a mixture and its components\. Second, the second targetbbenters the conditional models as an appended feature and is never encoded: in the regression checks it is continuous, and in the classification checks it is a categorical column encoded exactly like any other, with the conditioning levels drawn from the training levels\.

#### Target pairs\.

The two targets are chosen so that they plausibly depend on each other given the remaining features, which is what makes the consistency checks non\-trivial: ifaaandbbwere conditionally independent given𝐱\\mathbf\{x\}, the conditional and the marginal head would coincide and both conditions would hold for trivial reasons\. For classification,Winepairs the quality grade with the wine color,Diamondsthe cut grade with the color grade,Carthe acceptability rating with the safety rating,Nurserythe admission recommendation with the child’s health status,Annealthe annealing grade of a steel coil with the steel grade it was rolled from,Phishingthe verdict on a website with the behavior of its server form handler,MICthe outcome of a myocardial infarction with whether aspirin was administered in the ICU,Customerwhether a consumer loan defaulted with the borrower’s education level,Creditthe good/bad risk verdict with the balance band of the applicant’s checking account, andMarketingwhether a customer accepted the last campaign with that customer’s marital status\. Where several candidate columns existed we took the one most strongly associated with the main target, measured by Cramér’sVVon the full table, subject to its class distribution not being so skewed thatp^​\(b∣𝐱\)\\hat\{p\}\(b\\mid\\mathbf\{x\}\)collapses onto a single class, which would make the mixture in \([C1](https://arxiv.org/html/2608.06004#S3.Ex1)\) trivially equal to one of its components\. For regression the pairs are chosen by correlation:Concretepairs compressive strength with water content \(ρ=−0\.29\\rho=\-0\.29\),Bostonthe median home value with the average number of rooms \(ρ=0\.70\\rho=0\.70\),Stocktwo correlated company prices \(ρ=0\.71\\rho=0\.71\),Kin8nmthe end\-effector distance with the first joint angle \(ρ=−0\.14\\rho=\-0\.14\),QSARthe acute fish toxicity of a molecule with its octanol\-water partition coefficient \(ρ=0\.65\\rho=0\.65\),Fiatthe asking price of a used car with its odometer reading \(ρ=−0\.86\\rho=\-0\.86\), andInsurancethe annual billed medical expenses with the patient’s body\-mass index \(ρ=0\.20\\rho=0\.20\)\. The continuous case carries a second requirement that has no analog in the categorical one:bbmust not be a deterministic function of the remaining features, or the atoms discretizingp^​\(b∣𝐱\)\\hat\{p\}\(b\\mid\\mathbf\{x\}\)all collapse onto one value and the mixture reduces to a single conditional\. We therefore also check thatbbis not perfectly predictable from𝐱\\mathbf\{x\}, which is what excludes tables from designed experiments, where the design variables determine one another exactly\.

#### Curated re\-uploads\.

Anneal,Phishing,MIC,Customer,CreditandMarketing, together with the regression tablesQSAR,FiatandInsurance, are the versions of these tables curated for the TabArena tabular\-ML benchmark, which is why they are addressed by numeric OpenML ID rather than by name and version: the curation fixes column types and level names that the older uploads leave ambiguous, and the IDs pin the curated snapshot\. They also broaden the classification suite along the two axes on which the original four are narrow:Anneal,MICandMarketingare wide \(3737,5151and2424features\) and mix numerical with categorical columns, andMICcarries an eight\-class target whose classes are heavily imbalanced \(86%86\\%of its rows fall in the majority class\), so the consistency gaps are measured over sharper as well as flatter predictive distributions\.Marketingis also the one table whose second target has levels too rare to be learned: three of the eight values ofMarital\_Statusoccur in fewer than four rows\. We keep them rather than merge them, since the model simply assigns them negligible mass inp^​\(b∣𝐱\)\\hat\{p\}\(b\\mid\\mathbf\{x\}\)and they contribute correspondingly little to the mixture\.

#### Wine\.

This is the only dataset that is not a single OpenML table: the red\-wine and white\-wine quality tables are concatenated and an additional binary columncolorrecords which table a row came from, which supplies the second target\. The two tables share their eleven physicochemical features but label their quality column differently: the red table stores the raw quality scores33–88, whereas the white table stores them index\-encoded as11–77, so the merged target has eight levels rather than the seven distinct quality scores of the underlying data\.

## Appendix CDiscretization of the Regression Checks

Section[4](https://arxiv.org/html/2608.06004#S4)states that the regression checks compare distributions on a shared2020\-cell grid \(marginalization\) and a shared128×128128\\times 128grid \(factorization\)\. This appendix gives the exact construction of those cells and, since the models expose their predictive distributions in three different parameterizations, how each head is reduced to the single interface the checks consume\. Every step below is a deterministic function of the fitted models: there is no sampling anywhere in the pipeline, so repeating a run reproduces the reported numbers exactly\.

#### Common interface: the quantile curve\.

The checks never touch a model’s internals\. Each head is reduced to a*quantile curve*: its inverse CDFF^−1\\hat\{F\}^\{\-1\}evaluated on one fixed ladder ofL=2,000L=2\{,\}000equally spaced levels running fromα1=2\.5×10−4\\alpha\_\{1\}=2\.5\\times 10^\{\-4\}toαL=0\.99975\\alpha\_\{L\}=0\.99975\. Consecutive levels therefore bracket5×10−45\\times 10^\{\-4\}of predicted mass, and at most2\.5×10−42\.5\\times 10^\{\-4\}is left beyond the ladder in each tail\. A head may return a slightly non\-monotone level sequence \(quantile crossing\); we take a running maximum along the ladder, which restores a valid inverse CDF and acts as the identity whenever the head is already monotone\. Everything downstream is a function of the pairs\(αℓ,qℓ\)\(\\alpha\_\{\\ell\},q\_\{\\ell\}\)alone, so all models enter through the same interface and no model\-specific tolerance or tuning is involved\.

#### Per\-model access toF^−1\\hat\{F\}^\{\-1\}\.

The heads parameterize the predictive distribution differently, and each is inverted in the way its own parameterization prescribes\.

- •TabPFNv2 and v3emit a full\-support bar distribution: a softmax over5,0005\{,\}000bars, piecewise\-constant within a bar, with half\-normal tails attached to the outermost two\. Quantiles come from the model’s own inversion of that CDF\. Because the density is available in closed form, it is also what the reported NLL uses directly, with no finite\-difference approximation\.
- •TabICLv2emits a quantile distribution:999999predicted knots at levelsk/1,000k/1\{,\}000fork=1,…,999k=1,\\dots,999, joined by a monotone spline in the interior and extended by generalized\-Pareto tails beyond the outermost knots\. Its quantile function is the head’s native output, so evaluating the ladder is a direct call; its density is recovered from the reciprocal slope of the quantile curve\.
- •TabDPTemits a softmax over2,0482\{,\}048equal\-width bins spanning a fixed interval of the*standardized*target space\. Its public predictor returns only the bin\-center expectation, which is a point prediction and cannot support a distributional check; we therefore read the bin weights off the regression head itself and invert the resulting piecewise\-linear CDF, mapping the bin edges back to raw target units with the same standardization the head applies to its training targets\. Its density, like TabPFN’s, is exact\.

A head’s native resolution never binds the comparison: the coarsest of the three, TabICLv2’s999999knots, is already some fifty times finer than the2020cells on whichTV\\mathrm\{TV\}is finally evaluated\.

#### The2020cells \(marginalization\)\.

The grid is constructed per test instance*and per model*, from that model’s own predictions on that instance, rather than from any fixed global range\. Fix an instance𝐱i\\mathbf\{x\}\_\{i\}, writeqdirq^\{\\mathrm\{dir\}\}for the quantile curve of the direct marginalp^​\(a∣𝐱i;𝒟−BA\)\\hat\{p\}\(a\\mid\\mathbf\{x\}\_\{i\};\\mathcal\{D\}^\{A\}\_\{\-B\}\), and writeq\(k\)q^\{\(k\)\}for the curve of the conditionalp^​\(a∣bk,𝐱i;𝒟A\)\\hat\{p\}\(a\\mid b\_\{k\},\\mathbf\{x\}\_\{i\};\\mathcal\{D\}^\{A\}\)at thekk\-th of theK=1,000K=1\{,\}000equal\-mass atoms\. The grid spans the union of allK\+1K\+1supports,

loi\\displaystyle\\mathrm\{lo\}\_\{i\}=min⁡\{q1dir,mink⁡q1\(k\)\},\\displaystyle=\\min\\big\\\{q^\{\\mathrm\{dir\}\}\_\{1\},\\ \\textstyle\\min\_\{k\}q^\{\(k\)\}\_\{1\}\\big\\\},hii\\displaystyle\\mathrm\{hi\}\_\{i\}=max⁡\{qLdir,maxk⁡qL\(k\)\},\\displaystyle=\\max\\big\\\{q^\{\\mathrm\{dir\}\}\_\{L\},\\ \\textstyle\\max\_\{k\}q^\{\(k\)\}\_\{L\}\\big\\\},that is, the smallestα1\\alpha\_\{1\}\-quantile and the largestαL\\alpha\_\{L\}\-quantile among every distribution entering the comparison\. It is cut intoG=20G=20cells of equal width by the edgeseg=loi\+\(g−1\)​\(hii−loi\)/Ge\_\{g\}=\\mathrm\{lo\}\_\{i\}\+\(g\-1\)\(\\mathrm\{hi\}\_\{i\}\-\\mathrm\{lo\}\_\{i\}\)/Gforg=1,…,G\+1g=1,\\dots,G\+1, with a10−910^\{\-9\}widening guarding the degenerate casehii≤loi\\mathrm\{hi\}\_\{i\}\\leq\\mathrm\{lo\}\_\{i\}of a head that predicts a single point\.

Each CDF is evaluated at these edges by linear interpolation of its own\(qℓ,αℓ\)\(q\_\{\\ell\},\\alpha\_\{\\ell\}\)pairs, clamped to0belowq1q\_\{1\}and to11aboveqLq\_\{L\}\. WritingF​\(eg\)F\(e\_\{g\}\)for those edge values, the cell masses are

m1\\displaystyle m\_\{1\}=F​\(e2\),\\displaystyle=F\(e\_\{2\}\),mg\\displaystyle m\_\{g\}=F​\(eg\+1\)−F​\(eg\),1<g<G,\\displaystyle=F\(e\_\{g\+1\}\)\-F\(e\_\{g\}\),\\qquad 1<g<G,mG\\displaystyle m\_\{G\}=1−F​\(eG\)\.\\displaystyle=1\-F\(e\_\{G\}\)\.The boundary cells thus absorb the mass lying outside the grid, including the≤2\.5×10−4\\leq 2\.5\\times 10^\{\-4\}per side left beyond the ladder, so both discretized distributions sum to exactly11and no mass is discarded\. The reported gap isTV=12​∑g=1G\|mgmix−mgdir\|\\mathrm\{TV\}=\\tfrac\{1\}\{2\}\\sum\_\{g=1\}^\{G\}\\lvert m^\{\\mathrm\{mix\}\}\_\{g\}\-m^\{\\mathrm\{dir\}\}\_\{g\}\\rvert\.

Two properties of this construction matter for reading the numbers\. The grid is*shared*: both sides are discretized on the same edges with the same folding rule, so the boundary treatment is common to the two sides and cannot by itself open a gap\. And the grid is*model\-specific*: because its endpoints are read off the model’s own predicted support, a cell has a different width for different models, datasets and instances\. This does not impair comparability, becauseTV\\mathrm\{TV\}always compares two distributions produced by the same model on the same grid and is unit\-free\.

#### The128×128128\\times 128cells \(factorization\)\.

Here the two axes are discretized separately, and each axis is spanned by its own marginal head alone: the129129aa\-edges are equally spaced across\[q1a,qLa\]\[q^\{a\}\_\{1\},q^\{a\}\_\{L\}\], theα1\\alpha\_\{1\}\- andαL\\alpha\_\{L\}\-quantiles ofp^​\(a∣𝐱i;𝒟−BA\)\\hat\{p\}\(a\\mid\\mathbf\{x\}\_\{i\};\\mathcal\{D\}^\{A\}\_\{\-B\}\), and thebb\-edges are built the same way fromp^​\(b∣𝐱i;𝒟−AB\)\\hat\{p\}\(b\\mid\\mathbf\{x\}\_\{i\};\\mathcal\{D\}^\{B\}\_\{\-A\}\)\. The conditionals do not widen the grid, since each is evaluated only to fill one line of the joint table\.

Each axis then yields both cell masses, folded at the boundaries exactly as above, and one representative value per cell\. The representative is placed by midpoint quadrature*in probability space*rather than at the geometric cell center: if a cell’s edges carry cumulative levelsLgL\_\{g\}andLg\+1L\_\{g\+1\}, its representative isF^−1​\(12​\(Lg\+Lg\+1\)\)\\hat\{F\}^\{\-1\}\\big\(\\tfrac\{1\}\{2\}\(L\_\{g\}\+L\_\{g\+1\}\)\\big\), read off the same quantile curve\. A cell straddling a peak of the density is thus represented by a point near that peak instead of by its geometric middle\.

Ordera→ba\\to bbuilds the joint one line at a time: for eachbb\-cell, with masswlw\_\{l\}and representativebl∗b^\{\*\}\_\{l\}, the conditionalp^​\(a∣bl∗,𝐱i;𝒟A\)\\hat\{p\}\(a\\mid b^\{\*\}\_\{l\},\\mathbf\{x\}\_\{i\};\\mathcal\{D\}^\{A\}\)is evaluated, converted toaa\-axis cell masses on theaa\-edges by the same folding rule, and scaled bywlw\_\{l\}; the assembled table sums to11\. Orderb→ab\\to ais built symmetrically over theaa\-axis and transposed onto the same grid, so that the two tables are indexed by identical\(a,b\)\(a,b\)cells\. The reported gap is the half\-sum of absolute differences over all1282128^\{2\}of them\.

## Appendix DPredictive Quality versus Consistency

[Figures˜5](https://arxiv.org/html/2608.06004#A4.F5)and[6](https://arxiv.org/html/2608.06004#A4.F6)plot each model’s classification accuracy against its factorization and marginalization TV gap, one panel per dataset;[Figures˜7](https://arxiv.org/html/2608.06004#A4.F7)and[8](https://arxiv.org/html/2608.06004#A4.F8)do the same for regression, plotting RMSE in place of accuracy\. Across all four figures, more accurate \(or lower\-RMSE\) models are not systematically more consistent within a dataset: the two quantities do not trade off, but neither does either one predict the other from a single panel\.

The dataset\-level view above averages over instances, which leaves open how the inconsistency is distributed over rows\.[Figure˜4](https://arxiv.org/html/2608.06004#A4.F4)resolves this at the level of single test rows\. For each instance we pair the negative log\-likelihood that the direct predictivep^​\(a∣𝐱;𝒟−BA\)\\hat\{p\}\(a\\mid\\mathbf\{x\};\\mathcal\{D\}^\{A\}\_\{\-B\}\)assigns to the true label with the TV distance between that predictive and the marginalized mixture on the right\-hand side of \([C1](https://arxiv.org/html/2608.06004#S3.Ex1)\)\.

Instances whose NLL is near zero are those on which the direct head is already close to a point mass on one class\. Mixing overbbcannot move a saturated predictive far, so the TV on such rows is bounded near zero for arithmetic reasons alone, independently of whether the model is coherent\. Keeping them would draw a diagonal band into the bottom\-left corner of every panel and manufacture the appearance of TV growing steadily with the loss, an artifact of saturation rather than evidence that hard instances are the inconsistent ones\. We therefore drop every instance with an NLL below0\.050\.05, i\.e\., every instance on which the model already assigns roughly95%95\\%or more to the correct class, and plot only the remainder\.

[Figure˜4](https://arxiv.org/html/2608.06004#A4.F4)shows that there is no correlation between the likelihood assigned to the correct label and marginalization consistency at the per\-row level\.

![Refer to caption](https://arxiv.org/html/2608.06004v1/x3.png)Figure 4:Per\-instance predictive loss versus marginalization\-consistency gap\. The horizontal axis is the NLL that the direct predictive assigns to the true label, the vertical axis the TV distance between that predictive and the mixture obtained by marginalizing over the intermediate targetbb; both axes are logarithmic\. Instances with an NLL below0\.050\.05\(about95%95\\%probability on the correct class\) are omitted\.![Refer to caption](https://arxiv.org/html/2608.06004v1/x4.png)Figure 5:Predictive accuracy versus factorization\-consistency TV \([Table˜2](https://arxiv.org/html/2608.06004#S4.T2)\), one panel per classification dataset and one marker per model\.![Refer to caption](https://arxiv.org/html/2608.06004v1/x5.png)Figure 6:Predictive accuracy versus marginalization\-consistency TV \([Table˜1](https://arxiv.org/html/2608.06004#S4.T1)\), one panel per classification dataset and one marker per model\.![Refer to caption](https://arxiv.org/html/2608.06004v1/x6.png)Figure 7:Predictive RMSE versus factorization\-consistency TV \([Table˜2](https://arxiv.org/html/2608.06004#S4.T2)\), one panel per regression dataset and one marker per model\.![Refer to caption](https://arxiv.org/html/2608.06004v1/x7.png)Figure 8:Predictive RMSE versus marginalization\-consistency TV \([Table˜1](https://arxiv.org/html/2608.06004#S4.T1)\), one panel per regression dataset and one marker per model\.
## Appendix EConsistency versus Number of Features

[Figures˜9](https://arxiv.org/html/2608.06004#A5.F9),[10](https://arxiv.org/html/2608.06004#A5.F10),[11](https://arxiv.org/html/2608.06004#A5.F11)and[12](https://arxiv.org/html/2608.06004#A5.F12)plot each run’s consistency gap against the number of featuresDDof its dataset, and show no evidence that wider tables are harder to stay consistent on: the pooled Spearman correlation betweenDDand TV is negative in three of the four settings \(classificationρ=−0\.59\\rho=\-0\.59for factorization andρ=−0\.75\\rho=\-0\.75for marginalization, regressionρ=−0\.50\\rho=\-0\.50for factorization\) and positive only for marginalization on regression \(ρ=\+0\.40\\rho=\+0\.40\)\.

![Refer to caption](https://arxiv.org/html/2608.06004v1/x8.png)Figure 9:Factorization\-consistency TV versus the number of featuresDDon the classification datasets, one marker per \(dataset, model\)\.![Refer to caption](https://arxiv.org/html/2608.06004v1/x9.png)Figure 10:Marginalization\-consistency TV versus the number of featuresDDon the classification datasets, one marker per \(dataset, model\) run\.![Refer to caption](https://arxiv.org/html/2608.06004v1/x10.png)Figure 11:Factorization\-consistency TV versus the number of featuresDDon the regression datasets, one marker per \(dataset, model\) run\.![Refer to caption](https://arxiv.org/html/2608.06004v1/x11.png)Figure 12:Marginalization\-consistency TV versus the number of featuresDDon the regression datasets, one marker per \(dataset, model\) run\.