TabPFN-MT: A Natively Multitask In-Context Learner for Tabular Data

# TabPFN-MT: A Natively Multitask In-Context Learner for Tabular Data
Source: [https://arxiv.org/html/2605.20234](https://arxiv.org/html/2605.20234)
Cormac Cureton McGill University Mila \- Quebec AI Institute Montreal, QC, Canada cormac\.cureton@mail\.mcgill\.ca &Narges Armanfard McGill University Mila \- Quebec AI Institute Montreal, QC, Canada narges\.armanfard@mcgill\.ca

###### Abstract

Prior\-Data Fitted networks \(PFNs\) have been very successful in tabular contexts, handling prediction tasks in context\. However, they are designed for single\-task inference, meaning that predicting several target values within a context requires repeated forward calls and precludes inter\-task information sharing\. We propose TabPFN\-MT, which is trained on an expanded multi\-target synthetic prior to capture inter\-task dependencies in context\. This model uses an expandedyy\-encoder and a shared decoder head to enable multitask in\-context learning and simultaneous inference\. The model is uniquely specialized for small\-to\-medium datasets by relying on in\-context learning rather than traditional gradient\-based training\. Within this regime \(averaging fewer than 1,000 samples\), extensive evaluations across 344 datasets demonstrate that TabPFN\-MT establishes a new state\-of\-the\-art for deep tabular multitask learning\. Furthermore, despite the inherent compute asymmetry of joint optimization, our model remains highly competitive with the latest state\-of\-the\-art single\-task ensembles\. Notably, on multitask datasets it achieves an overall Accuracy rank of 4\.89, the highest average rank among all models tested\. Crucially, TabPFN\-MT delivers this highly competitive performance while reducing the inference cost forTTtasks from𝒪​\(T\)\\mathcal\{O\}\(T\)to𝒪​\(1\)\\mathcal\{O\}\(1\)forward passes, offering a massive computational efficiency improvement for multi\-target tabular applications\.

## 1Introduction

Tabular data is common across a wide range of domains, making it an important modality for the introduction of machine learning \(ML\) into real\-world applications\. In particular, there are many cases where datasets have several related outputs of interest\. For instance, a medical trial may have multiple key readouts, or in finance, it may be valuable to jointly model an asset’s return and volatility\.

Despite the prevalence of multi\-target problems, deep multitask learning \(MTL\) for tabular data has predominantly focused on massive datasets, such as e\-commerce and recommender systems\. In critical low\-data regimes, however, single\-task models like gradient\-boosted decision trees \(GBDTs\) remain the standard\. Applying these single\-task architectures to multi\-target problems requires fitting independent models for every target\. This approach inherently prevents inter\-task information sharing \(precluding positive transfer\) and multiplies the computational cost by the number of targetsTT\.

Recently, TabPFN has emerged as a leading model in low\-data tabular domains, leveraging in\-context learning \(ICL\) to tackle small datasets with a higher\-capacity model\(Hollmannet al\.,[2025](https://arxiv.org/html/2605.20234#bib.bib1)\)\. However, current tabular Prior\-Data Fitted Networks \(PFNs\) are constrained to single\-task inference\. Thus, predictingTTtargets requiresTTindependent forward passes, leading to a massive waste of computation, re\-calculating similar representations of the same tabular dataset\.

Simultaneously inferring multiple label distributions and modelling their interactions based on limited context is a substantially harder problem than standard scalar ICL\. However, evidence from Natural Language Processing demonstrates that pretrained foundation models are capable of emergent multitask ICL\(Xionget al\.,[2025](https://arxiv.org/html/2605.20234#bib.bib20)\)\. We hypothesize that this capability can be translated to the tabular domain\. Because a PFN can attend across the entire dataset context \(both features and targets\), it should be able to learn useful shared representations to support multiple simultaneous predictions\. This approach requires only a single forward pass for multitask prediction, opening the possibility for positive transfer and improved predictions overall\.

In this work, we introduceTabPFN\-MT, the first tabular foundation model natively pretrained for multitask in\-context learning\. Our core contributions are:

1. 1\.Multitask Extension of PFN Paradigm:This is the first MTL model to extend the PFN framework to tackle problems with a variable number of targets through joint inference\. Crucially, this enables amortized inference: a single pretrained model can be applied to a wide range of datasets with varying feature and target dimensionality without dataset\-specific training, instead leveraging ICL during inference\.
2. 2\.Multitask Architecture:To support this flexibility, TabPFN\-MT uses a dynamicyy\-encoder and a shared projection decoder\. The encoder scales inputs depending on the number of tasks in a dataset and uses dynamic zero\-padding to maintain consistent dimensions\. The decoder generates outputs in a single inference pass, which are then sliced into per\-task logit vectors\.
3. 3\.Synthetic Prior DesignWe demonstrate that pretraining this architecture on datasets generated by a single, complex Structural Causal Model \(SCM\) provides the necessary spectrum of task correlations to successfully teach a transformer to leverage shared representations in context\. This prior is sampled symmetrically allowing for stable training without loss balancing techniques common in other multitask works\.
4. 4\.Efficient State\-of\-the\-Art Low\-Data MTL:Through extensive evaluation across 344 datasets in the small\-to\-medium data regime \(<5,000<5,000samples\), TabPFN\-MT establishes a new state\-of\-the\-art for tabular MTL\. Furthermore, it achieves predictive performance competitive with compute\-heavy single\-task ensembles, while its architecture reduces computational overhead by reducing multi\-target inference cost from𝒪​\(T\)\\mathcal\{O\}\(T\)to𝒪​\(1\)\\mathcal\{O\}\(1\)forward passes\.

To support open science and future work building on these contributions, details about the future release of source code are included in[Appendix˜A](https://arxiv.org/html/2605.20234#A1)\.

The paper is organized as follows:[Section˜2](https://arxiv.org/html/2605.20234#S2)reviews prior work\.[Section˜3](https://arxiv.org/html/2605.20234#S3)details the multitask prior and TabPFN\-MT architecture\.[Section˜4](https://arxiv.org/html/2605.20234#S4)presents our comprehensive evaluations and comparisons to baselines, and[Section˜5](https://arxiv.org/html/2605.20234#S5)concludes with limitations and future directions\.

## 2Related Work

##### Standard Tabular Models

While deep learning has come to dominate ML applications across other modalities and domains, GBDTs remain the primary baseline for tabular data due to their strong handling of axis\-aligned features\(Chen and Guestrin,[2016](https://arxiv.org/html/2605.20234#bib.bib3); Keet al\.,[2017](https://arxiv.org/html/2605.20234#bib.bib18); Prokhorenkovaet al\.,[2018](https://arxiv.org/html/2605.20234#bib.bib6)\)\. However, their additive ensemble structure limits the applicability of multitask learning\. GBDT methods tend to treat each task as a separate objective for optimization, using different trees or leaves for each target\. This means that the models do not form shared representations across tasks and there is not a mechanism to share information across the targets\. Additionally, while GBDT models tend to require fewer parameters and offer strong out\-of\-the\-box baselines, achieving state\-of\-the\-art performance still requires rigorous hyperparameter optimization \(HPO\)\(McElfreshet al\.,[2023](https://arxiv.org/html/2605.20234#bib.bib21)\)\.

##### Tabular Deep Learning

While traditional ML approaches tend to operate on the raw data, models like SAINT\(Somepalliet al\.,[2021](https://arxiv.org/html/2605.20234#bib.bib5)\), FT\-Transformer\(Gorishniyet al\.,[2021](https://arxiv.org/html/2605.20234#bib.bib4)\), and TabNet\(Arik and Pfister,[2021](https://arxiv.org/html/2605.20234#bib.bib23)\)use attention mechanisms to create inter\-feature representations\. These models develop more rich internal representations but remain single task and single dataset; they must be trained from scratch for each new dataset and target\.

There have been several works that apply MTL to large tabular datasets\. Multi\-Gate Mixture\-of\-Experts \(MMoE\)\(Maet al\.,[2018](https://arxiv.org/html/2605.20234#bib.bib8)\), Progressive Layered Extraction \(PLE\)\(Tanget al\.,[2020](https://arxiv.org/html/2605.20234#bib.bib9)\), and Shared and Task\-specific EMbeddings \(STEM\)\(Suet al\.,[2024](https://arxiv.org/html/2605.20234#bib.bib7)\)all use a mixture of experts \(MoE\) approach with variations to mitigate negative transfer between tasks\. In contrast, MultiTab introduced a multitask masked attention mechanism to moderate the competition between tasks and improve multi\-target performance\(Sinodinoset al\.,[2026b](https://arxiv.org/html/2605.20234#bib.bib47)\)\. All of these MTL models are trained on the target dataset and the architectures must be fixed to the exact number of tasks seen during training\. In contrast, this paper looks to develop a more flexible model that leverages in\-context learning to infer task structure at inference time with less available data\.

##### Tabular Prior\-Data Fitted Networks

In low\-data regimes, PFNs have been very successful, pretraining on synthetic datasets and then adapting to target datasets through in\-context learning\(Mülleret al\.,[2022](https://arxiv.org/html/2605.20234#bib.bib10)\)\. TabPFN has achieved state\-of\-the\-art performance across a range of tabular benchmarks following this paradigm\(Hollmannet al\.,[2023](https://arxiv.org/html/2605.20234#bib.bib2),[2025](https://arxiv.org/html/2605.20234#bib.bib1); Grinsztajnet al\.,[2026](https://arxiv.org/html/2605.20234#bib.bib40)\)\. Subsequent works have looked to improve PFN scalability via linear attention\(Zenget al\.,[2025](https://arxiv.org/html/2605.20234#bib.bib22)\)and enhance pretraining with real\-world data\(Maet al\.,[2025](https://arxiv.org/html/2605.20234#bib.bib24)\)\.

All of these tabular PFNs have assumed single\-task, scalar labels, an assumption that breaks down in multitask settings\. Using a single\-task model for multitask problems requires distinct inferences for each target, leading to redundant calculation and not allowing for shared information between the tasks\.Sinodinoset al\.\([2026a](https://arxiv.org/html/2605.20234#bib.bib34)\)investigated adapting single\-task TabPFN to a multitask setting via multitask domain\-specific fine\-tuning\. In this work, we take a complementary approach, investigating whether a PFN can learn multitask in\-context inference through natively multitask pretraining with a modified prior\.

## 3Methodology

![Refer to caption](https://arxiv.org/html/2605.20234v1/x1.png)\(a\)Synthetic multi\-target data generation from a single underlying Structural Causal Model \(SCM\)\.
![Refer to caption](https://arxiv.org/html/2605.20234v1/x2.png)\(b\)The TabPFN\-MT architecture, with an expanded y\-encoder and decoder to handle multiple targets\.

Figure 1:The TabPFN\-MT Framework\.The model relies on a synthetic multitask prior generated via highly sparse Multi\-Layer Perceptrons \(MLPs\) simulating causal directed acyclic graphs \(DAGs\)\. During inference, the architecture leverages an expandedyy\-encoder and a dynamically sliced decoder head to perform simultaneous multitask in\-context learning over a shared Transformer backbone\.An overview of the complete TabPFN\-MT framework, detailing both the synthetic data generation process and the model architecture, is illustrated in[Figure˜1](https://arxiv.org/html/2605.20234#S3.F1)\.

### 3\.1Multitask In\-Context Learning

Mülleret al\.\([2022](https://arxiv.org/html/2605.20234#bib.bib10)\)established that PFNs make predictions in context via approximate Bayesian inference\. That is, given a labelled training datasetDtrainD\_\{\\text\{train\}\}and a test pointxtestx\_\{\\text\{test\}\}, the pretrained PFNqθq\_\{\\theta\}approximates the Posterior Predictive Distribution \(PPD\) for the label,p​\(ytest∣xtest,Dtrain\)p\(y\_\{\\text\{test\}\}\\mid x\_\{\\text\{test\}\},D\_\{\\text\{train\}\}\)\. In prior PFN models, the model predicts a single scalar target per query,qθ​\(ytest∣xtest,Dtrain\)≈p​\(ytest∣xtest,Dtrain\)q\_\{\\theta\}\(y\_\{\\text\{test\}\}\\mid x\_\{\\text\{test\}\},D\_\{\\text\{train\}\}\)\\approx p\(y\_\{\\text\{test\}\}\\mid x\_\{\\text\{test\}\},D\_\{\\text\{train\}\}\)\(Hollmannet al\.,[2023](https://arxiv.org/html/2605.20234#bib.bib2),[2025](https://arxiv.org/html/2605.20234#bib.bib1)\)\. In our multitask extension, the same PPD approximation underlies the model, but the context is expanded such thatDtrain=\{\(xi,𝐲i\)\}i=1ND\_\{\\text\{train\}\}=\\\{\(x\_\{i\},\\mathbf\{y\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}\. The target becomes a vector𝐲i=\(y\(1\),y\(2\),…,y\(T\)\)\\mathbf\{y\}\_\{i\}=\(y^\{\(1\)\},y^\{\(2\)\},\\dots,y^\{\(T\)\}\)corresponding to theTTdistinct tasks present in a given datasetDtrainD\_\{\\text\{train\}\}\. Our model prioritizes simultaneous inference for all targets, thus the model approximates the joint distribution of these targets as conditionally independent given the shared network representation\.

Let𝐡test\\mathbf\{h\}\_\{\\text\{test\}\}denote the final hidden state of the query point output by the transformer backbone, summarizing the labelled contextDtrainD\_\{\\text\{train\}\}and the query featuresxtestx\_\{\\text\{test\}\}\. The shared Multi\-Layer Perceptron \(MLP\) decoder maps this representation to a joint logit vector𝐥=MLPθhead​\(𝐡test\)\\mathbf\{l\}=\\text\{MLP\}\_\{\\theta\_\{\\text\{head\}\}\}\(\\mathbf\{h\}\_\{\\text\{test\}\}\)\.

The information sharing between tasks occurs natively within the hidden layers of this shared MLP, allowing the network to leverage learned correlations between tasks to shape the joint logit vector\. The static output𝐥\\mathbf\{l\}is then dynamically sliced into task\-specific logit vectors𝐥\(t\)∈ℝCmax\\mathbf\{l\}^\{\(t\)\}\\in\\mathbb\{R\}^\{C\_\{\\text\{max\}\}\}, whereCmaxC\_\{\\text\{max\}\}is the maximum number of classes per task\. The joint PPD is thus factorized at the output layer, where each marginal probability is parameterized by its respective logit slice:

qθ​\(𝐲test∣xtest,Dtrain\)=∏t=1Tqθ​\(ytest\(t\)∣𝐥\(t\)\)=∏t=1TSoftmax​\(𝐥\(t\)\)ytest\(t\)\.q\_\{\\theta\}\(\\mathbf\{y\}\_\{\\text\{test\}\}\\mid x\_\{\\text\{test\}\},D\_\{\\text\{train\}\}\)=\\prod\_\{t=1\}^\{T\}q\_\{\\theta\}\(y\_\{\\text\{test\}\}^\{\(t\)\}\\mid\\mathbf\{l\}^\{\(t\)\}\)=\\prod\_\{t=1\}^\{T\}\\text\{Softmax\}\(\\mathbf\{l\}^\{\(t\)\}\)\_\{y\_\{\\text\{test\}\}^\{\(t\)\}\}\.\(1\)

### 3\.2Multitask Structural Causal Model Prior

To generate the synthetic datasets used during the prior\-fitting phase, we build upon the SCM prior introduced in the original TabPFN\(Hollmannet al\.,[2023](https://arxiv.org/html/2605.20234#bib.bib2); Mülleret al\.,[2022](https://arxiv.org/html/2605.20234#bib.bib10)\)\. Datasets are generated by sampling a random directed acyclic graph \(DAG\) representing an SCM, propagating noise variables through non\-linear deterministic functions, and selecting a subset of nodes to act as observed featuresXXand a single targetYY\. The dataset is then partitioned into trainingDtrain=\(Xtrain,Ytrain\)D\_\{\\text\{train\}\}=\(X\_\{\\text\{train\}\},Y\_\{\\text\{train\}\}\)and test setsDtest=\(Xtest,Ytest\)D\_\{\\text\{test\}\}=\(X\_\{\\text\{test\}\},Y\_\{\\text\{test\}\}\)\. This approach yields diverse datasets with complex, conditionally dependent features driven by forward and backward causation\. We adopt this foundation but extend the data\-generation process to simulate multi\-target tabular environments\. Instead of a scalar target, the multitask SCM must sample a multi\-target matrix𝐘\\mathbf\{Y\}\.

A critical requirement for a robust multitask prior is the ability to generate datasets with varying degrees of inter\-task correlation\. If tasks are perfectly correlated, multitask learning is trivial; if they are entirely independent, models are susceptible to negative transfer\. In practice, these SCMs are parameterized as MLPs, where causal DAG structures are simulated by applying high sparsity masks to the network weights \([Figure˜1\(a\)](https://arxiv.org/html/2605.20234#S3.F1.sf1)\)\. The complexity of the task relationships is modulated by varying the depth, width, sparsity bounds, and the number of underlying MLPs sampled per dataset\. After testing multiple configurations for sampling multiple targets \(detailed in[Appendix˜B](https://arxiv.org/html/2605.20234#A2)\), we found that using a single, shared, high\-complexity SCM for the prior yielded the best downstream model\.

Using a large underlying SCM means that in some cases features and targets will be sampled close together \(yielding high relatedness\) whereas in other cases they will be very distinct \(yielding low relatedness\)\. Intuitively, this variation should lead to a model which is robust to different strengths of task relationships\. In all cases, there does exist a shared underlying causal structure between all features and targets which should incentivize the model to form shared representations to capture positive transfer\. In this work, we restrict our focus to multi\-target classification \(binary and multiclass\) supporting classification up to 10 classes per target\. We leave heterogeneous combinations of continuous and categorical targets as an interesting direction for future work\.

### 3\.3Multitask Transformer

Following the original TabPFN\(Hollmannet al\.,[2023](https://arxiv.org/html/2605.20234#bib.bib2)\), our model processes tabular data as a set of rows, utilizing self\-attention to route information between the labelled contextDtrainD\_\{\\text\{train\}\}and unlabelled queryxtestx\_\{\\text\{test\}\}\. To support multitask in\-context learning, the transformer backbone remains unmodified; all architectural changes are isolated to the boundaries of the network: the inputyy\-encoder and the output decoder head\. During inference, we similarly adopt their 32\-permutation ensembling strategy to reduce positional bias \(see[Appendix˜C](https://arxiv.org/html/2605.20234#A3)for details\)\.

#### 3\.3\.1Expandedyy\-Encoder

In a standard TabPFN, the target encoder projects a scalar label into the shared embedding space\. To accommodate our multitask formulation, we expand this to ingest the full target vector𝐲i\\mathbf\{y\}\_\{i\}\([Figure˜1\(b\)](https://arxiv.org/html/2605.20234#S3.F1.sf2)\)\. A primary challenge of in\-context multitask learning is handling datasets with a variable number of tasksTTup to an architectural maximumTmaxT\_\{\\text\{max\}\}\.

To resolve this without altering the underlying network dimensions, ouryy\-encoder employs a fixed\-width base projection preceded by a dynamic scaling and padding module\. Given an input target vector of lengthT≤TmaxT\\leq T\_\{\\text\{max\}\}, the vector is first multiplied by a magnitude\-preserving scaling factor \(Tmax/TT\_\{\\text\{max\}\}/T\)\. This critical scaling step ensures that the overall variance and magnitude of the network activations remain stable, regardless of how many tasks are present in the context\. The scaled vector is subsequently zero\-padded to the fixed maximum dimensionTmaxT\_\{\\text\{max\}\}before being projected into the model’sdembd\_\{\\text\{emb\}\}\-dimensional working space\.

#### 3\.3\.2Expanded Decoder Head

Similarly, the decoder head must be expanded to output predictions for all potential tasks simultaneously\. We utilize a single, shared Multi\-Layer Perceptron \(MLP\)\. The decoder consists of two linear layers separated by a GELU activation, projecting the final hidden representation of the query point𝐡test\\mathbf\{h\}\_\{\\text\{test\}\}through a hidden dimensiondhidd\_\{\\text\{hid\}\}to a static output dimensionnoutn\_\{\\text\{out\}\}\.

Crucially, this decoder design is chosen instead of other paradigms common in MTL models such as task\-specific routing or autoregressive decoding\. The introduction of task\-specific heads would impose an assumption of consistent dataset structure, with a constant number of targets and systematic relationships between them\. In our ICL paradigm, the architecture must accommodate arbitrary task dimensionality and different inter\-task relationships, thus there is no basis to train task\-specific heads\. Autoregressive decoding would allow for a variable numbers of targets but would introduce an artificial task ordering and degrade inference to𝒪​\(T\)\\mathcal\{O\}\(T\)\.

In our configuration,noutn\_\{\\text\{out\}\}is defined by the product of the architectural maximums:Tmax×CmaxT\_\{\\text\{max\}\}\\times C\_\{\\text\{max\}\}, whereCmaxC\_\{\\text\{max\}\}is the maximum number of classes per task\. For example, withTmax=5T\_\{\\text\{max\}\}=5andCmax=10C\_\{\\text\{max\}\}=10, the decoder maps to a 50\-dimensional logit vector\. As formalized in[Section˜3\.1](https://arxiv.org/html/2605.20234#S3.SS1), this static output vector is then dynamically sliced during the loss calculation to isolate the specific logits for theTTvalid tasks present in that dataset, while ignoring the outputs corresponding to the padded dimensions\. These maximums are critical in maintaining strict dimensional consistency that enables efficient, parallelized, batched pretraining across synthetic datasets with different structures\.

### 3\.4Training Objective and Configuration

The training objective remains aligned with prior PFNs: minimizing the cross\-entropy on held\-out test setsDtest⊂DD\_\{\\text\{test\}\}\\subset D\. To accommodate varying numbers of tasks per dataset, the model dynamically slices the output dimensions to match the target\. The loss is calculated independently only for theTTvalid tasks present in the current dataset context\. To ensure training stability across different datasets, the total loss is the average of the negative log\-likelihoods across these valid tasks for all test points:

ℒPFN​\(θ\)=𝔼D∼p​\(D\)​𝔼\(xtest,𝐲test\)∼Dtest​\[1T​∑t=1T−log⁡qθ​\(ytest\(t\)∣𝐥\(t\)\)\]\.\\mathcal\{L\}\_\{\\text\{PFN\}\}\(\\theta\)=\\mathbb\{E\}\_\{D\\sim p\(D\)\}\\mathbb\{E\}\_\{\(x\_\{\\text\{test\}\},\\mathbf\{y\}\_\{\\text\{test\}\}\)\\sim D\_\{\\text\{test\}\}\}\\left\[\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\-\\log q\_\{\\theta\}\(y\_\{\\text\{test\}\}^\{\(t\)\}\\mid\\mathbf\{l\}^\{\(t\)\}\)\\right\]\.\(2\)
In our synthetic multi\-target training setup, there is no need for gradient interventions which are common in MTL deep learning\(Cipollaet al\.,[2018](https://arxiv.org/html/2605.20234#bib.bib31); Chenet al\.,[2018](https://arxiv.org/html/2605.20234#bib.bib32); Yuet al\.,[2020](https://arxiv.org/html/2605.20234#bib.bib33)\)\. All targets are sampled symmetrically thus there are no systemic differences in task scale, noise, or difficulty across the synthetic datasets\. Thus, simple uniform averaging of cross\-entropies is sufficient for stable training\.

Among nine tested configurations varying embedding dimension and number of layers, the best\-performing backbone had just 8\.1M parameters; notably fewer than TabPFN v1’s 25\.8M\(Hollmannet al\.,[2023](https://arxiv.org/html/2605.20234#bib.bib2)\)\. This compression suggests training may be approaching a*complexity ceiling*of the synthetic prior, and the reduced parameter count directly benefits inference speed and memory footprint, furthering the efficiency gains discussed in[Section˜4\.3](https://arxiv.org/html/2605.20234#S4.SS3)\. Full configuration details are in[Appendix˜D](https://arxiv.org/html/2605.20234#A4)\.

TabPFN\-MT is trained for 200 epochs on 3\.28M synthetic datasets using Adam\(Kingma and Ba,[2014](https://arxiv.org/html/2605.20234#bib.bib30)\); complete training details are provided in[Appendix˜E](https://arxiv.org/html/2605.20234#A5)\.

## 4Experiments

In this section, we evaluate the predictive performance and computational efficiency of TabPFN\-MT\. We first detail the datasets, baselines, and metrics used in our evaluation, followed by a comprehensive analysis of the results\.

### 4\.1Experimental Setup

To ensure a rigorous and fair comparison, our evaluation framework encompasses a wide variety of tabular environments and state\-of\-the\-art models\. The specific details of our setup are outlined below\.

#### 4\.1\.1Real\-world Datasets

To comprehensively evaluate TabPFN\-MT across diverse data regimes, we curated a benchmark of 344 real\-world datasets across four distinct categories\. First, to establish a Native Multitask baseline, we curated multi\-target classification datasets from OpenML, filtering out instances of extreme class imbalance \(Imbalance Ratio\>16\>16\) and restricting bounds to a maximum of 5,000 samples, 100 features, and 5 targets\. Second, we utilized a subset of Standard Single\-Task \(T=1T=1\) OpenML datasets drawn from the validation suite established byHollmannet al\.\([2023](https://arxiv.org/html/2605.20234#bib.bib2)\)\. Third, to robustly evaluate performance across varying target correlations, we constructed Feature\-Derived Multitask datasets by programmatically repurposing input features as new joint target vectors from established single\-target benchmarks\. Finally, we evaluated two Subsampled Large\-Scale datasets, AliExpress\(Liet al\.,[2020](https://arxiv.org/html/2605.20234#bib.bib36); Xiet al\.,[2021](https://arxiv.org/html/2605.20234#bib.bib37)\)and ACS Income\(Dinget al\.,[2021](https://arxiv.org/html/2605.20234#bib.bib35); Maet al\.,[2018](https://arxiv.org/html/2605.20234#bib.bib8)\), randomly subsampled down to\{500,1000,2500,5000\}\\\{500,1000,2500,5000\\\}instances across five seeds to adapt these massive benchmarks to the low\-data regime targeted by our model\. Summary statistics and implementation details for all categories of datasets are provided in[Appendix˜F](https://arxiv.org/html/2605.20234#A6)\.

#### 4\.1\.2Baselines

##### Single\-Task Models

To fully contextualize the performance of TabPFN\-MT, we compare it against state\-of\-the\-art multitask models as well as independent ensembles of single\-task models\. For the single\-task baselines, we train a separate model for each target, a standard practice in MTL\(Goodfellowet al\.,[2016](https://arxiv.org/html/2605.20234#bib.bib38)\)\. For our single\-task baselines, we evaluate a standard MLP, alongside Random Forest \(RF\)\(Breiman,[2001](https://arxiv.org/html/2605.20234#bib.bib42)\), XGBoost\(Chen and Guestrin,[2016](https://arxiv.org/html/2605.20234#bib.bib3)\), and CatBoost\(Prokhorenkovaet al\.,[2018](https://arxiv.org/html/2605.20234#bib.bib6)\), as gradient\-boosted decision trees have long been held as the strongest defaults for tabular data\. For deep single\-task baselines, we include TabTransformer \(Tab\-T\)\(Huanget al\.,[2020](https://arxiv.org/html/2605.20234#bib.bib39)\), FT\-Transformer \(FT\-T\)\(Gorishniyet al\.,[2021](https://arxiv.org/html/2605.20234#bib.bib4)\), and SAINT\(Somepalliet al\.,[2021](https://arxiv.org/html/2605.20234#bib.bib5)\)\. We also evaluate against TabPFN v1\(Hollmannet al\.,[2023](https://arxiv.org/html/2605.20234#bib.bib2)\), whose architecture TabPFN\-MT is based upon, and the most recent TabPFN release, v2\.6, which builds upon the changes described in the TabPFN v2\.5 report\(Grinsztajnet al\.,[2026](https://arxiv.org/html/2605.20234#bib.bib40)\)\.

##### Multitask Models

For multitask baselines, we compare against large\-scale deep models, including MoE MLP architectures—MMOE\(Maet al\.,[2018](https://arxiv.org/html/2605.20234#bib.bib8)\), PLE\(Tanget al\.,[2020](https://arxiv.org/html/2605.20234#bib.bib9)\), and STEM\(Suet al\.,[2024](https://arxiv.org/html/2605.20234#bib.bib7)\)—as well as the transformer\-based MultiTab \(MTT\)\(Sinodinoset al\.,[2026b](https://arxiv.org/html/2605.20234#bib.bib47)\)\. It should be noted that the small dataset sizes that this paper examines are outside of the regime that these models were developed for\.

To ensure a rigorous comparison, all baselines undergo hyperparameter optimization \(HPO\)\. Details on baseline implementations and the search spaces and HPO processes are detailed in[Appendix˜G](https://arxiv.org/html/2605.20234#A7)\.

#### 4\.1\.3Metrics

To evaluate classification performance, we report Accuracy, F1 Score, and Area Under the Receiver Operating Characteristic Curve \(ROC AUC\) aggregated across allTTtargets\. Since our model encounters both binary and multiclass classification problems, we use both binary and multiclass formulations of those metrics\.

To assess benefits of multitask performance for a modelmm, we use multitask gain \(Δm\\Delta\_\{m\}\) fromManiniset al\.\([2019](https://arxiv.org/html/2605.20234#bib.bib45)\)\.Δm\\Delta\_\{m\}is calculated as the average change in performance for a measureMMcompared to a single\-task baselinebbacross all tasksi=1,…,Ti=1,\\dots,T\. For the baseline we use a single\-task MLP,

Δm=1T​∑i=1TMm,i−Mb,iMb,i\.\\Delta\_\{m\}=\\frac\{1\}\{T\}\\sum\_\{i=1\}^\{T\}\\frac\{M\_\{m,i\}\-M\_\{b,i\}\}\{M\_\{b,i\}\}\.\(3\)
Because our metrics are strictly “higher is better,” a positiveΔm\\Delta\_\{m\}indicates that the multitask model outperforms the single\-task baseline\.

Detailed formulations on multiclass metric handling, our cross\-validation setup, and our statistical significance testing pipeline \(Friedman and Nemenyi tests\) are provided in[Appendix˜H](https://arxiv.org/html/2605.20234#A8)\.

### 4\.2Predictive Performance

TabPFN\-MT ranks first among all evaluated multitask models across every metric and closely approaches the performance of state\-of\-the\-art single\-task ensembles \([Table˜1](https://arxiv.org/html/2605.20234#S4.T1)\)\. The Critical Difference \(CD\) diagram for Accuracy \([Figure˜2](https://arxiv.org/html/2605.20234#S4.F2)\) places our model in the top\-performing clique, indicating statistical equivalence with the strongest single\-task baselines\. Crucially, TabPFN\-MT is theonlymultitask model in this top clique, achieving this performance with𝒪​\(1\)\\mathcal\{O\}\(1\)computational complexity regarding the number of tasks, compared to the𝒪​\(T\)\\mathcal\{O\}\(T\)scaling required by the single\-task baselines \([Table˜11](https://arxiv.org/html/2605.20234#A10.T11)\)\. Additional CD diagrams in[Appendix˜I](https://arxiv.org/html/2605.20234#A9)confirm this trend: TabPFN\-MT remains the highest\-ranked MTL model overall, belonging to the top clique for both F1 and ROC AUC\.

To better characterize where TabPFN\-MT excels, we stratify predictive performance by dataset origin \(detailed in[Appendix˜I](https://arxiv.org/html/2605.20234#A9)\)\. We observe that while deep MTL baselines \(e\.g\., Multitask MLP and MultiTab\) show relative improvement on the subsampled Large MTL datasets due to the higher sample counts, they still fail to outperform strong single\-task models even in this regime\. In contrast, TabPFN\-MT demonstrates exceptional performance on the Derived MTL datasets, achieving the highest overall Accuracy \(0\.526\) and outperforming all other multitask baselines by a wide margin across all metrics\. Because these datasets are generated by repurposing existing input features as joint targets, they inherently possess strong, structured causal relationships\. This environment closely aligns with the inductive bias of our SCM synthetic prior, where multiple targets are sampled from the same dense causal graph\. Consequently, TabPFN\-MT is highly effective at identifying these inter\-task dependencies in context and leveraging them for positive cross\-task transfer\.

Conversely, joint inference introduces predictive trade\-offs, most notably reflected in the macro\-averaged F1 scores\. While TabPFN\-MT consistently leads the multitask baselines, it occasionally trails the absolute best single\-task models \(specifically TabPFN v2\.6 and Random Forest\) in F1 performance, particularly on the Native MTL and Large MTL splits\. Single\-task architectures dedicate their entire functional capacity to optimizing a single, isolated decision boundary, allowing them to more precisely capture minority classes in imbalanced scenarios\. In contrast, TabPFN\-MT must distribute its shared representation and decoder capacity across multiple targets simultaneously \(explored in[Section˜I\.1](https://arxiv.org/html/2605.20234#A9.SS1)\)\. As observed in our target\-scaling ablation, this shared capacity creates an information bottleneck that can slightly degrade performance on complex, heavily imbalanced, or weakly correlated tasks\.

Table 1:Overall comparison across key metrics with all models included\. Top\-level columns separate Single\-Target and Multi\-Target dataset evaluation views\. Within each column,boldmarks the best MTL model value andunderlinedtext marks the best STL model value \(ties share the same formatting\)\.![Refer to caption](https://arxiv.org/html/2605.20234v1/cd_all_accuracy.png)Figure 2:TabPFN\-MT is in the top clique for Accuracy and is the highest\-ranked multitask model\. The diagram displays the average rank of each model across all 344 datasets evaluated, with lower ranks indicating better performance \(further left\)\. Thick horizontal lines connect groups of models \(cliques\) that do not exhibit statistically significant performance differences from one another based on a Nemenyi post\-hoc test \(p<0\.05p<0\.05\)\. While TabPFN v2\.6 achieves the best overall average rank, our proposed TabPFN\-MT performs competitively within the same top\-tier statistical clique and significantly outperforms all other multitask baselines\.
### 4\.3Computational Efficiency

To systematically evaluate the behaviour of TabPFN\-MT as the target space grows, we benchmark the total floating\-point operations \(FLOPs\) required across increasing numbers of tasks\. While comparing FLOPs provides a rigorous, hardware\-agnostic measure of computational cost, tree\-based models like GBDT baselines cannot be captured in this analysis\. Detailed information regarding the FLOP counting methodology and wall\-clock comparisons that include GBDT baselines are provided in[Appendix˜J](https://arxiv.org/html/2605.20234#A10)\.

We use the MultiTab framework\(Sinodinoset al\.,[2026b](https://arxiv.org/html/2605.20234#bib.bib47)\)to generate synthetic tabular datasets\. To benchmark computational cost, datasets are standardized to 1,000 samples, 50 features, a noise level of 0\.01, and polynomial degrees ranging from 1 to 3\. We vary the number of targets from 1 to 5 and fix task correlation at 0\.4\.

As illustrated in[Figure˜3](https://arxiv.org/html/2605.20234#S4.F3), single\-target baselines \(e\.g\., standard TabPFN variants, SAINT, and FT\-Transformer\) incur a linear computational cost \(𝒪​\(T\)\\mathcal\{O\}\(T\)\) as the number of targets increases\. These models require independent training and inference processes for each target\. In contrast, multitask models like TabPFN\-MT handle targets simultaneously allowing for constant computational cost \(𝒪​\(1\)\\mathcal\{O\}\(1\)\) even as the number of targets increases\.

Furthermore, because TabPFN\-MT extends the PFN paradigm, it does not require dataset\-specific training, achieving highly competitive computational costs that fall below other multitask baselines\. The lightweight model backbone described above in[Section˜3\.4](https://arxiv.org/html/2605.20234#S3.SS4)also means that TabPFN\-MT’s cost is orders of magnitude below single\-task TabPFNs \(even in the single\-target domain\)\.

It is important to note that this dataset\-level analysis does not capture the initial pretraining phase of TabPFN\-MT, which required approximately 4 hours across eight RTX 5000 GPUs\. However, unlike traditional baselines that require costly hyperparameter optimization and iterative training for every newly encountered dataset, TabPFN\-MT adapts entirely in context\. Consequently, this one\-time pretraining investment is quickly amortized across downstream applications, ultimately providing a massive reduction in cumulative compute for multi\-target tabular environments\.

![Refer to caption](https://arxiv.org/html/2605.20234v1/flops_scaling_by_targets.png)Figure 3:Computational cost scaling across architectures\. Total FLOPs \(log scale\) from training and inference are evaluated on synthetic datasets containing 1 to 5 targets\. Single\-target architectures exhibit𝒪​\(T\)\\mathcal\{O\}\(T\)scaling, requiring linearly more compute for each additional target\. In contrast, our proposed model, TabPFN\-MT \(solid purple line\), natively infers all targets in a single forward pass, achieving𝒪​\(1\)\\mathcal\{O\}\(1\)scaling and maintaining constant computational efficiency\. Note that this measurement excludes the cost of HPO which is relevant for all models except for the PFNs\.

## 5Discussion and Conclusion

In this work we introduce TabPFN\-MT, a novel tabular foundation model that establishes a new state\-of\-the\-art for deep tabular models in the small\-to\-medium sized data regime\. On the datasets tested, our work consistently outperforms existing MTL models and achieves predictive parity \(Accuracy, F1, ROC AUC\) with top\-performing ensembles of single\-task models, including TabPFN v2\.6 and GBDTs\. Crucially, it achieves this level of performance while reducing the multi\-target inference cost from𝒪​\(T\)\\mathcal\{O\}\(T\)to𝒪​\(1\)\\mathcal\{O\}\(1\)compared to the application of target\-specific single\-task models\. Furthermore, TabPFN\-MT uses in\-context adaptation rather than gradient\-based for dataset\-specific adaptation, thus avoiding the negative transfer risk which is a problem in traditional MTL approaches\. The pretraining phase uses synthetic data which does not have consistent gradient conflicts\. During the forward pass, the transformer architecture has sufficient capacity in the multi\-head attention mechanism to dynamically attend to features and labels in the support set, avoiding destructive interference but still offering the potential for positive cross\-task transfer\.

##### Limitations

While foundation models inherently require an initial pretraining investment, TabPFN\-MT trains in just four hours across eight GPUs\. This represents a fraction of the compute typically required for foundation models\. This one\-time cost is amortized across a wide range of downstream applications which offer high performance without dataset\-specific finetuning or hyperparameter optimization\.

Currently the model is scoped for classification problems with a maximum of 5 tasks and 10 classes but the same approach could be extended to regression following the approach with a piece\-wise distribution taken byHollmannet al\.\([2025](https://arxiv.org/html/2605.20234#bib.bib1)\)\. The task and class count limits could be scaled in future if motivated by downstream applications\.

Furthermore, our work finds that a low\-parameter \(8\.1M\) model variant can outperform larger models\. This suggests that this model had sufficient capacity to map the prior’s complexity\. This suggests that further development of this class of models will not come from simply adding layers, but instead from engineering more complex proxy distributions which still transfer to real\-world data\. This finding is aligned with recent work that has begun to introduce real\-world data into the pretraining of tabular foundation models likeMaet al\.\([2025](https://arxiv.org/html/2605.20234#bib.bib24)\)andGrinsztajnet al\.\([2026](https://arxiv.org/html/2605.20234#bib.bib40)\)\.

##### Societal Impact

TabPFN\-MT does not require the same compute\-expensive hyperparameter optimization phase as other tabular methods; thus its adoption could help reduce environmental impact of tabular deep learning\. As tabular data is frequently found in high\-sensitivity domains \(e\.g\., healthcare, finance\), practitioners should be mindful of historical biases in the data used for predictions\. Because the model uses in\-context adaptation to make predictions based on limited data, it will propagate biases captured in a target dataset into predictions\. The rigorous curation of unbiased support sets is crucial to ensure responsible and equitable outcomes when applying this work\.

## References

- TabNet: Attentive Interpretable Tabular Learning\.Proceedings of the AAAI Conference on Artificial Intelligence35\(8\),pp\. 6679–6687\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/16826),[Document](https://dx.doi.org/10.1609/aaai.v35i8.16826)Cited by:[§2](https://arxiv.org/html/2605.20234#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Breiman \(2001\)Random Forests\.Machine Learning45\(1\),pp\. 5–32\(en\)\.External Links:ISSN 1573\-0565,[Link](https://doi.org/10.1023/A:1010933404324),[Document](https://dx.doi.org/10.1023/A%3A1010933404324)Cited by:[§4\.1\.2](https://arxiv.org/html/2605.20234#S4.SS1.SSS2.Px1.p1.1)\.
- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei \(2020\)Language Models are Few\-Shot Learners\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 1877–1901\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html)Cited by:[Appendix E](https://arxiv.org/html/2605.20234#A5.p2.1)\.
- T\. Chen and C\. Guestrin \(2016\)XGBoost: A Scalable Tree Boosting System\.InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,KDD ’16,New York, NY, USA,pp\. 785–794\.External Links:ISBN 978\-1\-4503\-4232\-2,[Link](https://dl.acm.org/doi/10.1145/2939672.2939785),[Document](https://dx.doi.org/10.1145/2939672.2939785)Cited by:[§2](https://arxiv.org/html/2605.20234#S2.SS0.SSS0.Px1.p1.1),[§4\.1\.2](https://arxiv.org/html/2605.20234#S4.SS1.SSS2.Px1.p1.1)\.
- Z\. Chen, V\. Badrinarayanan, C\. Lee, and A\. Rabinovich \(2018\)GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks\.InProceedings of the 35th International Conference on Machine Learning,pp\. 794–803\(en\)\.External Links:ISSN 2640\-3498,[Link](https://proceedings.mlr.press/v80/chen18a.html)Cited by:[§3\.4](https://arxiv.org/html/2605.20234#S3.SS4.p2.1)\.
- R\. Cipolla, Y\. Gal, and A\. Kendall \(2018\)Multi\-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics\.In2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition,Salt Lake City, UT, USA,pp\. 7482–7491\(en\)\.External Links:ISBN 978\-1\-5386\-6420\-9,[Link](https://ieeexplore.ieee.org/document/8578879/),[Document](https://dx.doi.org/10.1109/CVPR.2018.00781)Cited by:[§3\.4](https://arxiv.org/html/2605.20234#S3.SS4.p2.1)\.
- J\. Demšar \(2006\)Statistical Comparisons of Classifiers over Multiple Data Sets\.J\. Mach\. Learn\. Res\.7,pp\. 1–30\.External Links:ISSN 1532\-4435,[Link](https://dl.acm.org/doi/10.5555/1248547.1248548)Cited by:[§H\.3](https://arxiv.org/html/2605.20234#A8.SS3.p1.1)\.
- F\. Ding, M\. Hardt, J\. Miller, and L\. Schmidt \(2021\)Retiring Adult: New Datasets for Fair Machine Learning\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 6478–6490\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2021/hash/32e54441e6382a7fbacbbbaf3c450059-Abstract.html)Cited by:[§F\.4](https://arxiv.org/html/2605.20234#A6.SS4.p1.1),[§4\.1\.1](https://arxiv.org/html/2605.20234#S4.SS1.SSS1.p1.3)\.
- I\. Goodfellow, Y\. Bengio, and A\. Courville \(2016\)Deep learning\.MIT Press\.Note:[http://www\.deeplearningbook\.org](http://www.deeplearningbook.org/)Cited by:[§4\.1\.2](https://arxiv.org/html/2605.20234#S4.SS1.SSS2.Px1.p1.1)\.
- Y\. Gorishniy, I\. Rubachev, V\. Khrulkov, and A\. Babenko \(2021\)Revisiting Deep Learning Models for Tabular Data\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 18932–18943\.External Links:[Link](https://proceedings.neurips.cc/paper/2021/hash/9d86d83f925f2149e9edb0ac3b49229c-Abstract.html)Cited by:[§2](https://arxiv.org/html/2605.20234#S2.SS0.SSS0.Px2.p1.1),[§4\.1\.2](https://arxiv.org/html/2605.20234#S4.SS1.SSS2.Px1.p1.1)\.
- L\. Grinsztajn, K\. Flöge, O\. Key, F\. Birkel, P\. Jund, B\. Roof, B\. Jäger, D\. Safaric, S\. Alessi, A\. Hayler, M\. Manium, R\. Yu, F\. Jablonski, S\. B\. Hoo, A\. Garg, J\. Robertson, M\. Bühler, V\. Moroshan, L\. Purucker, C\. Cornu, L\. C\. Wehrhahn, A\. Bonetto, B\. Schölkopf, S\. Gambhir, N\. Hollmann, and F\. Hutter \(2026\)TabPFN\-2\.5: Advancing the State of the Art in Tabular Foundation Models\.arXiv\(en\)\.Note:arXiv:2511\.08667 \[cs\]External Links:[Link](http://arxiv.org/abs/2511.08667),[Document](https://dx.doi.org/10.48550/arXiv.2511.08667)Cited by:[§2](https://arxiv.org/html/2605.20234#S2.SS0.SSS0.Px3.p1.1),[§4\.1\.2](https://arxiv.org/html/2605.20234#S4.SS1.SSS2.Px1.p1.1),[§5](https://arxiv.org/html/2605.20234#S5.SS0.SSS0.Px1.p3.1)\.
- D\. J\. Hand and R\. J\. Till \(2001\)A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems\.Machine Learning45\(2\),pp\. 171–186\(en\)\.External Links:ISSN 1573\-0565,[Link](https://doi.org/10.1023/A:1010920819831),[Document](https://dx.doi.org/10.1023/A%3A1010920819831)Cited by:[§H\.1](https://arxiv.org/html/2605.20234#A8.SS1.p1.1)\.
- N\. Hollmann, S\. Müller, K\. Eggensperger, and F\. Hutter \(2023\)TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=cp5PvcI6w8_)Cited by:[Appendix B](https://arxiv.org/html/2605.20234#A2.p7.1),[Appendix C](https://arxiv.org/html/2605.20234#A3.p1.3),[Appendix E](https://arxiv.org/html/2605.20234#A5.p1.1),[Appendix E](https://arxiv.org/html/2605.20234#A5.p2.1),[§F\.2](https://arxiv.org/html/2605.20234#A6.SS2.p1.1),[§G\.2](https://arxiv.org/html/2605.20234#A7.SS2.p2.1),[§2](https://arxiv.org/html/2605.20234#S2.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2605.20234#S3.SS1.p1.9),[§3\.2](https://arxiv.org/html/2605.20234#S3.SS2.p1.5),[§3\.3](https://arxiv.org/html/2605.20234#S3.SS3.p1.3),[§3\.4](https://arxiv.org/html/2605.20234#S3.SS4.p3.1),[§4\.1\.1](https://arxiv.org/html/2605.20234#S4.SS1.SSS1.p1.3),[§4\.1\.2](https://arxiv.org/html/2605.20234#S4.SS1.SSS2.Px1.p1.1)\.
- N\. Hollmann, S\. Müller, L\. Purucker, A\. Krishnakumar, M\. Körfer, S\. B\. Hoo, R\. T\. Schirrmeister, and F\. Hutter \(2025\)Accurate predictions on small data with a tabular foundation model\.Nature637\(8045\),pp\. 319–326\(en\)\.External Links:ISSN 1476\-4687,[Link](https://www.nature.com/articles/s41586-024-08328-6),[Document](https://dx.doi.org/10.1038/s41586-024-08328-6)Cited by:[§1](https://arxiv.org/html/2605.20234#S1.p3.2),[§2](https://arxiv.org/html/2605.20234#S2.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2605.20234#S3.SS1.p1.9),[§5](https://arxiv.org/html/2605.20234#S5.SS0.SSS0.Px1.p2.1)\.
- X\. Huang, A\. Khetan, M\. Cvitkovic, and Z\. Karnin \(2020\)TabTransformer: Tabular Data Modeling Using Contextual Embeddings\.arXiv\.Note:arXiv:2012\.06678 \[cs\]External Links:[Link](http://arxiv.org/abs/2012.06678),[Document](https://dx.doi.org/10.48550/arXiv.2012.06678)Cited by:[§4\.1\.2](https://arxiv.org/html/2605.20234#S4.SS1.SSS2.Px1.p1.1)\.
- G\. Ke, Q\. Meng, T\. Finley, T\. Wang, W\. Chen, W\. Ma, Q\. Ye, and T\. Liu \(2017\)LightGBM: A Highly Efficient Gradient Boosting Decision Tree\.InAdvances in Neural Information Processing Systems,Vol\.30\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html)Cited by:[§2](https://arxiv.org/html/2605.20234#S2.SS0.SSS0.Px1.p1.1)\.
- D\. P\. Kingma and J\. Ba \(2014\)Adam: a method for stochastic optimization\.CoRRabs/1412\.6980\.External Links:[Link](https://api.semanticscholar.org/CorpusID:6628106)Cited by:[Appendix E](https://arxiv.org/html/2605.20234#A5.p2.1),[§3\.4](https://arxiv.org/html/2605.20234#S3.SS4.p4.1)\.
- P\. Li, R\. Li, Q\. Da, A\. Zeng, and L\. Zhang \(2020\)Improving Multi\-Scenario Learning to Rank in E\-commerce by Exploiting Task Relationships in the Label Space\.InProceedings of the 29th ACM International Conference on Information & Knowledge Management,CIKM ’20,New York, NY, USA,pp\. 2605–2612\.External Links:ISBN 978\-1\-4503\-6859\-9,[Link](https://dl.acm.org/doi/10.1145/3340531.3412713),[Document](https://dx.doi.org/10.1145/3340531.3412713)Cited by:[§F\.4](https://arxiv.org/html/2605.20234#A6.SS4.p1.1),[§4\.1\.1](https://arxiv.org/html/2605.20234#S4.SS1.SSS1.p1.3)\.
- J\. Ma, Z\. Zhao, X\. Yi, J\. Chen, L\. Hong, and E\. H\. Chi \(2018\)Modeling Task Relationships in Multi\-task Learning with Multi\-gate Mixture\-of\-Experts\.InProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,KDD ’18,New York, NY, USA,pp\. 1930–1939\.External Links:ISBN 978\-1\-4503\-5552\-0,[Link](https://doi.org/10.1145/3219819.3220007),[Document](https://dx.doi.org/10.1145/3219819.3220007)Cited by:[§F\.4](https://arxiv.org/html/2605.20234#A6.SS4.p1.1),[§2](https://arxiv.org/html/2605.20234#S2.SS0.SSS0.Px2.p2.1),[§4\.1\.1](https://arxiv.org/html/2605.20234#S4.SS1.SSS1.p1.3),[§4\.1\.2](https://arxiv.org/html/2605.20234#S4.SS1.SSS2.Px2.p1.1)\.
- J\. Ma, V\. Thomas, R\. Hosseinzadeh, A\. Labach, J\. C\. Cresswell, K\. Golestan, G\. Yu, A\. L\. Caterini, and M\. Volkovs \(2025\)TabDPT: Scaling Tabular Foundation Models on Real Data\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,\(en\)\.External Links:[Link](https://openreview.net/forum?id=pIZxEOZCId)Cited by:[§2](https://arxiv.org/html/2605.20234#S2.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2605.20234#S5.SS0.SSS0.Px1.p3.1)\.
- K\. Maninis, I\. Radosavovic, and I\. Kokkinos \(2019\)Attentive single\-tasking of multiple tasks\.2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 1851–1860\.External Links:[Link](https://api.semanticscholar.org/CorpusID:121100839)Cited by:[§4\.1\.3](https://arxiv.org/html/2605.20234#S4.SS1.SSS3.p2.6)\.
- D\. McElfresh, S\. Khandagale, J\. Valverde, V\. P\. C\., G\. Ramakrishnan, M\. Goldblum, and C\. White \(2023\)When do neural nets outperform boosted trees on tabular data?\.InProceedings of the 37th International Conference on Neural Information Processing Systems,Red Hook, NY, USA,pp\. 76336–76369\.Cited by:[§2](https://arxiv.org/html/2605.20234#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Müller, N\. Hollmann, S\. P\. Arango, J\. Grabocka, and F\. Hutter \(2022\)Transformers Can Do Bayesian Inference\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=KSugKcbNf9)Cited by:[§2](https://arxiv.org/html/2605.20234#S2.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2605.20234#S3.SS1.p1.9),[§3\.2](https://arxiv.org/html/2605.20234#S3.SS2.p1.5)\.
- L\. Prokhorenkova, G\. Gusev, A\. Vorobev, A\. V\. Dorogush, and A\. Gulin \(2018\)CatBoost: unbiased boosting with categorical features\.InAdvances in Neural Information Processing Systems,Vol\.31\.External Links:[Link](https://papers.nips.cc/paper_files/paper/2018/hash/14491b756b3a51daac41c24863285549-Abstract.html)Cited by:[§2](https://arxiv.org/html/2605.20234#S2.SS0.SSS0.Px1.p1.1),[§4\.1\.2](https://arxiv.org/html/2605.20234#S4.SS1.SSS2.Px1.p1.1)\.
- R\. Shwartz\-Ziv and A\. Armon \(2022\)Tabular data: Deep learning is not all you need\.Information Fusion81,pp\. 84–90\.External Links:ISSN 1566\-2535,[Link](https://www.sciencedirect.com/science/article/pii/S1566253521002360),[Document](https://dx.doi.org/10.1016/j.inffus.2021.11.011)Cited by:[§G\.2](https://arxiv.org/html/2605.20234#A7.SS2.p2.1)\.
- D\. Sinodinos, B\. Nikpour, J\. Y\. Wei, S\. Sinha, X\. Ma, K\. Rehman, S\. Yue, and N\. Armanfard \(2026a\)Multitask\-Informed Prior for In\-Context Learning on Tabular Data: Application to Steel Property Prediction\.arXiv\.Note:arXiv:2603\.22738 \[cs\]External Links:[Link](http://arxiv.org/abs/2603.22738),[Document](https://dx.doi.org/10.48550/arXiv.2603.22738)Cited by:[§2](https://arxiv.org/html/2605.20234#S2.SS0.SSS0.Px3.p2.1)\.
- D\. Sinodinos, J\. Y\. Wei, and N\. Armanfard \(2026b\)MultiTab: A Scalable Foundation for Multitask Learning on Tabular Data\.Proceedings of the AAAI Conference on Artificial Intelligence40\(30\),pp\. 25499–25507\.Cited by:[§G\.1](https://arxiv.org/html/2605.20234#A7.SS1.p1.1),[§2](https://arxiv.org/html/2605.20234#S2.SS0.SSS0.Px2.p2.1),[§4\.1\.2](https://arxiv.org/html/2605.20234#S4.SS1.SSS2.Px2.p1.1),[§4\.3](https://arxiv.org/html/2605.20234#S4.SS3.p2.1)\.
- G\. Somepalli, M\. Goldblum, A\. Schwarzschild, C\. B\. Bruss, and T\. Goldstein \(2021\)SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre\-Training\.arXiv\.Note:arXiv:2106\.01342 \[cs\]External Links:[Link](http://arxiv.org/abs/2106.01342),[Document](https://dx.doi.org/10.48550/arXiv.2106.01342)Cited by:[§2](https://arxiv.org/html/2605.20234#S2.SS0.SSS0.Px2.p1.1),[§4\.1\.2](https://arxiv.org/html/2605.20234#S4.SS1.SSS2.Px1.p1.1)\.
- L\. Su, J\. Pan, X\. Wang, X\. Xiao, S\. Quan, X\. Chen, and J\. Jiang \(2024\)STEM: Unleashing the Power of Embeddings for Multi\-Task Recommendation\.Proceedings of the AAAI Conference on Artificial Intelligence38\(8\),pp\. 9002–9010\(en\)\.Note:Number: 8External Links:ISSN 2374\-3468,[Link](https://ojs.aaai.org/index.php/AAAI/article/view/28749),[Document](https://dx.doi.org/10.1609/aaai.v38i8.28749)Cited by:[§2](https://arxiv.org/html/2605.20234#S2.SS0.SSS0.Px2.p2.1),[§4\.1\.2](https://arxiv.org/html/2605.20234#S4.SS1.SSS2.Px2.p1.1)\.
- H\. Tang, J\. Liu, M\. Zhao, and X\. Gong \(2020\)Progressive Layered Extraction \(PLE\): A Novel Multi\-Task Learning \(MTL\) Model for Personalized Recommendations\.InProceedings of the 14th ACM Conference on Recommender Systems,RecSys ’20,New York, NY, USA,pp\. 269–278\.External Links:ISBN 978\-1\-4503\-7583\-2,[Link](https://doi.org/10.1145/3383313.3412236),[Document](https://dx.doi.org/10.1145/3383313.3412236)Cited by:[§2](https://arxiv.org/html/2605.20234#S2.SS0.SSS0.Px2.p2.1),[§4\.1\.2](https://arxiv.org/html/2605.20234#S4.SS1.SSS2.Px2.p1.1)\.
- D\. Xi, Z\. Chen, P\. Yan, Y\. Zhang, Y\. Zhu, F\. Zhuang, and Y\. Chen \(2021\)Modeling the Sequential Dependence among Audience Multi\-step Conversions with Multi\-task Learning in Targeted Display Advertising\.InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining,KDD ’21,New York, NY, USA,pp\. 3745–3755\.External Links:ISBN 978\-1\-4503\-8332\-5,[Link](https://dl.acm.org/doi/10.1145/3447548.3467071),[Document](https://dx.doi.org/10.1145/3447548.3467071)Cited by:[§F\.4](https://arxiv.org/html/2605.20234#A6.SS4.p1.1),[§4\.1\.1](https://arxiv.org/html/2605.20234#S4.SS1.SSS1.p1.3)\.
- Z\. Xiong, Z\. Cai, J\. Cooper, A\. Ge, V\. Papageorgiou, Z\. Sifakis, A\. Giannou, Z\. Lin, L\. Yang, S\. Agarwal, G\. Chrysos, S\. Oymak, K\. Lee, and D\. Papailiopoulos \(2025\)Everything Everywhere All at Once: LLMs can In\-Context Learn Multiple Tasks in Superposition\.InForty\-second International Conference on Machine Learning,\(en\)\.External Links:[Link](https://openreview.net/forum?id=5hZCK4Wbex)Cited by:[§1](https://arxiv.org/html/2605.20234#S1.p4.1)\.
- T\. Yu, S\. Kumar, A\. Gupta, S\. Levine, K\. Hausman, and C\. Finn \(2020\)Gradient Surgery for Multi\-Task Learning\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 5824–5836\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/3fe78a8acf5fda99de95303940a2420c-Abstract.html)Cited by:[§3\.4](https://arxiv.org/html/2605.20234#S3.SS4.p2.1)\.
- Y\. Zeng, T\. Dinh, W\. Kang, and A\. C\. Mueller \(2025\)TabFlex: Scaling Tabular Learning to Millions with Linear Attention\.InForty\-second International Conference on Machine Learning,\(en\)\.External Links:[Link](https://openreview.net/forum?id=d60cmFf89H)Cited by:[§2](https://arxiv.org/html/2605.20234#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix AReproducibility Statement

To ensure the reproducibility of our results, we will provide source code, hyperparameter configurations, and instructions for generating the synthetic data in the future upon the formal publication of this work\. We emphasize that this approach is accessible to the broader academic community\. A complete pretraining run of the multitask model takes just over four hours on a single node equipped with eight GPUs \(NVIDIA A5000\)\.

## Appendix BMultitask Prior Details

To systematically determine the optimal synthetic data generation strategy, we employed a multi\-stage Bayesian hyperparameter optimization pipeline using BOHB \(Bayesian Optimization and Hyperband\) through Weights and Biases\. This search was conducted in three successive phases to isolate the effects of prior structure and training dynamics\.

First, we conducted a broad exploratory sweep over the structural paradigms, allowing the prior to sample varying numbers of underlying SCMs to map the relationship between task correlation and model capacity \([Table˜2](https://arxiv.org/html/2605.20234#A2.T2)\)\. Second, having identified promising prior configurations, we performed a dedicated search over the transformer’s training hyperparameters to ensure stable optimization during joint inference \([Table˜4](https://arxiv.org/html/2605.20234#A2.T4)\)\. Finally, guided by our empirical findings that fewer SCMs yield superior downstream performance, we executed a highly focused search constraining the prior strictly to a single, high\-complexity SCM to fine\-tune the causal graph parameters \([Table˜3](https://arxiv.org/html/2605.20234#A2.T3)\)\.

Table 2:Search bounds for initial sweep over prior meta\-hyperparamters with multiple underlying SCMs\.Table 3:Search bounds for secondary sweep over prior meta\-hyperparamters with single SCMs\.Table 4:Search bounds for training hyperparameter sweep\.To model this spectrum of task relatedness, we initially formalized the data\-generating process under three distinct structural paradigms:

- •Single Shared SCM:A single causal graph is sampled\. AllTTtargets and all features are drawn from the observed nodes of this unified graph, simulating domains with high latent feature sharing and highly correlated targets\.
- •TTDistinct SCMs:TTindependent SCMs are sampled\. Each targety\(t\)y^\{\(t\)\}is drawn from its own distinct graph, while the feature setXXis constructed by pooling observed nodes across allTTgraphs\. This simulates distinct tasks that coincidentally share a feature space, yielding low inter\-task correlation\.
- •Distributednn\-SCMs:A hybrid approach wheren∼U​\(1,T\)n\\sim U\(1,T\)SCMs are sampled\. The features and targets are randomly distributed across thesennunderlying graphs, creating clusters of related tasks within the same dataset\.

To evaluate the proposed structural paradigms, we conducted an extensive hyperparameter search over the prior configurations\. We initially hypothesized that sampling from a higher number of distinct SCMs \(n≫1n\\gg 1\) would be necessary to generate sufficiently diverse, low\-correlation tasks\. However, our empirical evaluations demonstrated a strong negative correlation between the number of underlying MLPs and overall multitask performance\.

The experiments revealed two primary failure modes when scaling the number of distinct SCMs\. First, forcing the prior to utilize multiple distinct MLPs led to severe computational bottlenecks during synthetic data generation; the complex graph construction and sampling processes starved the GPU, significantly reducing training efficiency\. Second, and more critically, datasets generated from many independent SCMs induced negative transfer during the transformer’s in\-context learning phase, degrading the model’s ability to form unified latent representations\.

Analysis of the top\-performing configurations showed that the model heavily favours a minimal number of shared SCMs with higher individual complexity\. Specifically, the optimal bounds restricted the prior to sampling between one and two underlying MLPs \(n∈\[1,2\]n\\in\[1,2\]\)\.

Subsequent evaluation of models trained with a prior with a single shared SCM revealed that it further improved model performance\. This is in line with the trend that priors with fewer underlying MLPs lead to better performance\. Interestingly, our HP sweep revealed that priors that favour with more complex sparse SCMs \(ie\., more layers and higher hidden dimensions\) than the parameters used in the original TabPFN work\[Hollmannet al\.,[2023](https://arxiv.org/html/2605.20234#bib.bib2)\]\. However, especially in the case of the number of layers, there are diminishing returns: increasing the number of layers does not monotonically increase resulting performance and increases the computational resources needed to generate synthetic data\. Intuitively, this prior configuration incentivizes modelling of a single underlying causal structure, although different features and targets may be sampled from very different parts of the structure\. The complete configuration of the top\-performing prior is included in[Table˜5](https://arxiv.org/html/2605.20234#A2.T5)\.

The final resolved configuration for our top\-performing prior \(detailed in[Table˜5](https://arxiv.org/html/2605.20234#A2.T5)\) reflects these structural insights\. To generate a sufficiently rich and varied multitask environment from a single causal graph, the prior relies on relatively deep \(five to eigth layers\) and wide \(up to 278 hidden dimensions\) MLPs\. Crucially, to ensure these networks simulate sparse causal DAGs rather than dense neural representations, the configuration employs heavy regularization, including high MLP dropout probabilities and block\-wise dropout\. The injection of diverse activation functions \(e\.g\., ELU, Tanh, Leaky ReLU\) further increases the non\-linear complexity of the generated functions\. By sampling datasets with a dynamically varying number of targets \(T∈\[1,5\]T\\in\[1,5\]\) from this single robust structure, the resulting prior forces the transformer to learn flexible, shared representations that generalize effectively to real\-world multitask tabular data\.

Table 5:Resolved prior configuration used for top\-performing training run\.
## Appendix CEnsembling

Following the methodology ofHollmannet al\.\[[2023](https://arxiv.org/html/2605.20234#bib.bib2)\], we employ an ensembling technique at inference time to improve predictive robustness and mitigate positional bias\. Rather than relying on a single forward pass, we average the model’s predictions over an ensemble of up to 32 data permutations\. These permutations are constructed using combinations of feature column rotations, class label rotations, and the application of a power transformation\. To prevent redundant evaluations, the number of ensemble members is strictly bounded to the maximum number of unique pre\-processing combinations,2​k​j2kj, wherekkis the number of features andjjis the number of classes\. Unless otherwise noted, all reported evaluations utilize this ensembling approach, which yields marginal but consistent performance improvements\.

## Appendix DModel Configuration and Scaling

The transformer backbone has four attention heads, pre\-normalization, and no dropout\. The feed\-forward hidden dimension is double the embedding dimension,dhid=2×dembd\_\{\\text\{hid\}\}=2\\times d\_\{\\text\{emb\}\}\.

We investigate a range of model configurations varying the embedding dimensiondemb∈\{256,384,512\}d\_\{\\text\{emb\}\}\\in\\\{256,384,512\\\}and the number of layersL∈\{9,12,15\}L\\in\\\{9,12,15\\\}\. We conduct a grid\-search across the nine configurations using the best prior from the prior configuration search \(which used the median model architecturedemb=384d\_\{\\text\{emb\}\}=384,L=12L=12\)\. For all model configurations, training spans 200 epochs so all architectures see the same number of training examples\. Based on the loss curves, training has plateaued for all models by this point\. We use the same scheduling, and the peak learning rate is adjusted logarithmically based on the number of parameters; details on both are described in[Appendix˜E](https://arxiv.org/html/2605.20234#A5)\. Full results are displayed below in[Figure˜4](https://arxiv.org/html/2605.20234#A4.F4)\.

![Refer to caption](https://arxiv.org/html/2605.20234v1/heatmap_accuracy.png)\(a\)Accuracy
![Refer to caption](https://arxiv.org/html/2605.20234v1/heatmap_f1.png)\(b\)F1 Score
![Refer to caption](https://arxiv.org/html/2605.20234v1/heatmap_roc_auc.png)\(c\)ROC AUC

Figure 4:Performance heatmaps for varying model configurations\. The grid search evaluates 9 distinct architectures modifying the embedding dimension \(dembd\_\{\\text\{emb\}\}\) and number of layers \(LL\)\. Across all 3 metrics, the ’Small Deep’ configuration \(demb=256d\_\{\\text\{emb\}\}=256,L=15L=15\) achieves the best performance\.
## Appendix ETraining Details

The training configuration is based on the hyperparameters used in TabPFN v1\[Hollmannet al\.,[2023](https://arxiv.org/html/2605.20234#bib.bib2)\]with some adjustments\. The synthetic datasets are sampled with 1,024 samples each and the evaluation position which separates the context and test points is sampled uniformly between position 0 and 1,000 for each dataset\. Our model can handle a maximum of 100 features, 5 targets, and up to 10 classes per target\.

The model is trained for 200 epochs of 128 batches each, with batch size 128 and gradient accumulation of 4\. Thus the model sees 3,276,800 synthetic datasets throughout training\. Training uses the Adam optimizer\[Kingma and Ba,[2014](https://arxiv.org/html/2605.20234#bib.bib30)\]with no weight decay and gradient clipping is applied with a maximum norm of 1\.0 to ensure stability\. The learning rate increases linearly over five epochs to a peak value of9\.379×10−49\.379\\times 10^\{\-4\}then decays following a cosine schedule\. The peak learning rate was scaled according to the total parameter count of the network, utilizing the logarithmic curve\-fit heuristic fromHollmannet al\.\[[2023](https://arxiv.org/html/2605.20234#bib.bib2)\], which approximates the optimal learning rates observed in large language models\[Brownet al\.,[2020](https://arxiv.org/html/2605.20234#bib.bib26)\]\.

Training uses mixed precision and flash attention to accelerate training and inference performance\. All models were implemented in PyTorch\. The models were trained using Distributed Data Parallel \(DDP\) across several NVIDIA RTX 5000 Ada \(32 GB\) GPUs\. Hardware usage for hyperparameter sweeps and training runs varied between one and eight GPUs depending on the task\. During training, the prior data is generated using a CPU with eight dedicated workers\.

## Appendix FEvaluation Dataset Details

TabPFN\-MT is evaluated using datasets from four distinct groups: Native Multitask, Single\-Task \(STL\), Feature\-Derived Multitask, and Subsampled Large\-Scale Multitask\. The coverage and sample size distributions for each category are detailed in[Table˜6](https://arxiv.org/html/2605.20234#A6.T6)\.

Table 6:Dataset coverage for model comparisons\. The MTL evaluation spans 283 datasets with a diverse range of sample sizes, focusing on the low\-data regime whereDDindicates the number of datasets\.### F\.1Native Multitask Datasets

Due to the lack of established multitask benchmarks in the small\-to\-medium data regime, we curated a collection of multi\-target classification datasets from OpenML\. We observed that many of these datasets exhibit extreme class imbalance, rendering them more akin to anomaly detection tasks than standard classification\. To address this, we filtered the collection to exclude datasets with a maximum Imbalance Ratio \(IR\) greater than 16, defined asIR=\|Majority Class\|\|Minority Class\|\\text\{IR\}=\\frac\{\|\\text\{Majority Class\}\|\}\{\|\\text\{Minority Class\}\|\}\. Furthermore, to align with our experimental scope, we restricted the final selection to datasets containing a maximum of 5,000 samples, 100 features, and five targets\.

### F\.2Standard Single\-Task Datasets

To evaluate performance on standard single\-task classification problems \(T=1T=1\), we utilized a subset of OpenML datasets drawn from the validation suite established byHollmannet al\.\[[2023](https://arxiv.org/html/2605.20234#bib.bib2)\]\.

### F\.3Feature\-Derived Multitask Datasets

To robustly evaluate model performance across a wider variety of domain structures and target correlations, we programmatically derived a suite of multi\-target datasets from established single\-target classification benchmarks\. For each base dataset, we dropped the original target and repurposed a subset of the input features to act as a new joint target vector𝐲\\mathbf\{y\}of lengthTT, whereT∈\{2,3,4,5\}T\\in\\\{2,3,4,5\\\}\.

To ensure diverse and representative task combinations, the transformation process alternates between selecting nominal and numeric features\.

- •Nominal featuresare extracted and treated directly as multiclass classification targets\.
- •Numeric featuresare cast into classification tasks via quantile discretization\. To introduce varying levels of task complexity, the number of quantile bins systematically cycles throughb∈\{3,4,5,6\}b\\in\\\{3,4,5,6\\\}\.

Crucially, once a feature is designated as a target, it is strictly removed from the input feature spaceXXto prevent trivial identity mapping\. Datasets lacking sufficient features to support the extraction while maintaining at least one input feature were skipped\. This methodology yields a challenging set of naturally occurring multitask environments where the generative relationships between the remaining features and the newly assigned targets are inherently complex and domain\-grounded\.

### F\.4Subsampled Large\-Scale Datasets

Additionally, we used two large scale multi\-target datasets which have been common to evaluate deep tabular models: AliExpress\[Liet al\.,[2020](https://arxiv.org/html/2605.20234#bib.bib36)\]using the processed version fromXiet al\.\[[2021](https://arxiv.org/html/2605.20234#bib.bib37)\]and ACS Income\[Dinget al\.,[2021](https://arxiv.org/html/2605.20234#bib.bib35)\]using the 2 target setup fromMaet al\.\[[2018](https://arxiv.org/html/2605.20234#bib.bib8)\]\. However, because these full datasets contain millions of samples they are not suitable for our model as is, thus we subsample randomly from the datasets down to\{500,1000,2500,5000\}\\\{500,1000,2500,5000\\\}samples\. We evaluate five random seeds for each size yielding 20 total subsampled datasets from each base dataset\.

## Appendix GBaseline Details

To ensure a rigorous, fair, and reproducible evaluation, this section details the specific implementations and optimization protocols used for all baseline models compared against TabPFN\-MT\.

### G\.1Baseline Implementations

We utilize PyTorch implementations fromSinodinoset al\.\[[2026b](https://arxiv.org/html/2605.20234#bib.bib47)\]for most deep models\. For XGBoost, CatBoost, and TabPFN v2\.6, we use their official libraries\. For TabPFN v1, we use a public implementation and checkpoint, as the original weights are no longer accessible via the official TabPFN library\.

### G\.2Baseline Hyperparameter Search

We conducted hyperparameter optimization for all baseline models and aggregated the results across 5 random seeds\. Given the massive computational scale of evaluating across 150 datasets, all models were optimized using a constrained budget of 25 random search trials via Optuna, with each trial scored using 3\-fold cross\-validation \(CV\)\. To prevent the neural architectures from overfitting their training folds, our PyTorch estimators dynamically reserved a10%10\\%validation fraction from within each training fold to enable early stopping \(patience = 10 epochs\)\. Tree\-based models, which are far less susceptible to epoch\-based overfitting, were trained on the full CV training folds\.

For XGBoost and CatBoost, we followHollmannet al\.\[[2023](https://arxiv.org/html/2605.20234#bib.bib2)\]and utilize the hyperparameter search spaces established byShwartz\-Ziv and Armon \[[2022](https://arxiv.org/html/2605.20234#bib.bib46)\]\. For Random Forest, we adopt a regularized search space, tuning from 50 to 300 estimators and restricting maximum depth from 3 to 15\. Finally, for deep learning baselines, we specifically designed the search spaces to favor lower architectural capacities and heavier regularization \(e\.g\., expanded ranges for dropout and weight decay\) to prevent catastrophic overfitting in these low\-data regimes\. Full details on hyperparameter search spaces are included below in[Table˜7](https://arxiv.org/html/2605.20234#A7.T7)\.

Table 7:Hyperparameter search spaces for tree\-based and deep learning baselines\. The bounds are adapted from the MultiTab framework\. Static parameters used during deep baseline training include an early stopping patience of 10 epochs and a validation fraction of 10%\.HyperparameterDistributionRange / ChoicesRandom ForestNumber of estimatorsUniform Int\[50,300\]\[50,300\]Max depthUniform Int\[3,15\]\[3,15\]Min samples splitUniform Int\[2,10\]\[2,10\]Min samples leafUniform Int\[1,5\]\[1,5\]Max featuresCategorical\{sqrt,log2,None\}\\\{\\text\{sqrt\},\\text\{log2\},\\text\{None\}\\\}XGBoostNumber of estimatorsUniform Int\[100,4000\]\[100,4000\]Learning rateLogUniform\[10−7,1\]\[10^\{\-7\},1\]Max depthUniform Int\[1,10\]\[1,10\]SubsampleUniform\[0\.2,1\.0\]\[0\.2,1\.0\]Colsample by treeUniform\[0\.2,1\.0\]\[0\.2,1\.0\]Colsample by levelUniform\[0\.2,1\.0\]\[0\.2,1\.0\]Min child weightLogUniform\[10−16,105\]\[10^\{\-16\},10^\{5\}\]Alpha \(L1 reg\.\)LogUniform\[10−16,102\]\[10^\{\-16\},10^\{2\}\]Lambda \(L2 reg\.\)LogUniform\[10−16,102\]\[10^\{\-16\},10^\{2\}\]GammaLogUniform\[10−16,102\]\[10^\{\-16\},10^\{2\}\]CatBoostLearning rateLogUniform\[10−5,1\]\[10^\{\-5\},1\]Random strengthUniform\[1,20\]\[1,20\]One\-hot max sizeUniform Int\[0,25\]\[0,25\]L2 leaf regLogUniform\[1,10\]\[1,10\]Bagging temperatureUniform\[0,1\]\[0,1\]Leaf estimation iterationsUniform Int\[1,20\]\[1,20\]Deep Learning Baselines\(MultiTab, MT\-MLP, MMoE, PLE, STEM, TabTransformer, SAINT, FT\-Transformer\)Maximum epochsUniform Int\[40,max\_epochs\]\[40,\\text\{max\\\_epochs\}\]Batch sizeCategorical\{128,256,512\}\\\{128,256,512\\\}Learning rateLogUniform\[10−4,5×10−3\]\[10^\{\-4\},5\\times 10^\{\-3\}\]Weight decayLogUniform\[10−7,10−3\]\[10^\{\-7\},10^\{\-3\}\]DropoutUniform\[0\.0,0\.4\]\[0\.0,0\.4\]Embedding dimensionCategorical\{16,32,64\}\\\{16,32,64\\\}Hidden dimensionCategorical\{64,128,256\}\\\{64,128,256\\\}Number of blocks \(layers\)Uniform Int\[2,4\]\[2,4\]Number of attention headsCategorical\{2,4,8\}\\\{2,4,8\\\}Feed\-forward hidden dimCategorical\{64,128,256\}\\\{64,128,256\\\}

## Appendix HExtended Evaluation Details

### H\.1Multiclass Metric Aggregation

The metric calculation in our experiments adapts dynamically depending on the cardinality of each target\. For binary classification tasks, we report the standard binary F1 score and binary ROC AUC\. For multiclass targets, we apply specific handling to ensure fair evaluation across varying and imbalanced class distributions: F1 scores are reported using a macro\-average to equally weight all classes, and ROC AUC is calculated using the One\-vs\-One \(OVO\) macro\-average approach, which computes the average AUC across all possible pairwise class combinations\[Hand and Till,[2001](https://arxiv.org/html/2605.20234#bib.bib43)\]\.

Because our architecture jointly predicts varying numbers of tasks, dataset\-level performance is evaluated by first computing these metrics independently for each of theTTtargets\. The overall performance for a given dataset is then reported as the unweighted mean of these individual target metrics\. In instances where a metric is mathematically undefined for a specific target, it is excluded from the dataset’s aggregated mean\.

### H\.2Cross\-Validation and Robustness

To ensure robustness against variances in data splitting and optimization, all baseline and proposed models undergo 5\-fold cross\-validation during evaluation\. This entire training and evaluation pipeline is repeated across three independent random seeds\. The final reported metrics for each dataset represent the mean performance across these repeated trials\.

### H\.3Statistical Significance Testing

To assess the statistical significance of our comparative results across a large corpus of datasets, we follow the non\-parametric statistical pipeline recommended byDemšar \[[2006](https://arxiv.org/html/2605.20234#bib.bib44)\]\. We first apply the Friedman test to reject the null hypothesis that all models perform equivalently\. Subsequently, we use the Nemenyi post\-hoc test to perform pairwise comparisons between models\. The results of these tests are visualized using Critical Difference \(CD\) diagrams, where models connected by a thick horizontal bar belong to the same top\-performing "clique" and are not statistically significantly different from one another at a significance level ofα=0\.05\\alpha=0\.05\.

## Appendix IExtended Results

This section provides the complete, granular breakdown of the experimental results introduced in[Section˜4](https://arxiv.org/html/2605.20234#S4)\. While the main text reports aggregated performance,[Table˜8](https://arxiv.org/html/2605.20234#A9.T8),[Table˜9](https://arxiv.org/html/2605.20234#A9.T9), and[Table˜10](https://arxiv.org/html/2605.20234#A9.T10)stratify the results across our distinct data regimes described above in[Appendix˜F](https://arxiv.org/html/2605.20234#A6): standard Single\-Task Learning \(STL\), Native MTL, Feature\-Derived MTL, and Subsampled Large MTL\.

For each metric, we report the average score alongside the average rank in the format ofvalue \(rank\)\. Stratifying the models at this level reveals how performance shifts depending on the dataset’s origin, scale, and inherent task correlation\. For example, while deep multitask baselines show improved competitiveness on the Subsampled Large MTL datasets, they generally struggle in the low\-data regimes\.

Conversely, TabPFN\-MT maintains robust performance across all multitask categories\. Notably, its strong performance on the Feature\-Derived MTL datasets—where targets share an explicit, inherent causal relationship with the inputs—validates our synthetic data generation strategy, demonstrating that the model successfully captures structural dependencies similar to those generated by our single Structural Causal Model \(SCM\) prior\.

Table 8:Average ROC\-AUC across dataset sources\. Each cell reports average metric value and average rank as value \(rank\)\. Within each source column,boldmarks the best value andunderlinedtext marks the second\-best value \(ties share the same formatting\)\. For ranks, lower is better\.Table 9:Average Accuracy across dataset sources\. Each cell reports average metric value and average rank as value \(rank\)\. Within each source column,boldmarks the best value andunderlinedtext marks the second\-best value \(ties share the same formatting\)\. For ranks, lower is better\.Table 10:Average F1 across dataset sources\. Each cell reports average metric value and average rank as value \(rank\)\. Within each source column,boldmarks the best value andunderlinedtext marks the second\-best value \(ties share the same formatting\)\. For ranks, lower is better\.![Refer to caption](https://arxiv.org/html/2605.20234v1/cd_all_f1.png)\(a\)F1 Score:TabPFN\-MT belongs to the top statistical clique and is the highest\-ranked multitask model\.
![Refer to caption](https://arxiv.org/html/2605.20234v1/cd_all_roc_auc.png)\(b\)ROC AUC:TabPFN\-MT ranks second overall and shares the top\-performing clique with TabPFN v2\.6\.

Figure 5:Critical difference diagrams for auxiliary metrics\.Consistent with accuracy results, TabPFN\-MT significantly outperforms all other evaluated multitask baselines across both F1 and ROC AUC\. Notably, for both metrics, our model is not significantly different from the top\-ranked baselines, placing it in the highest statistical tier\.### I\.1Predictive Performance vs\. Target Count

![Refer to caption](https://arxiv.org/html/2605.20234v1/mtl_gain_vs_targets_roc_auc.png)Figure 6:Effect of task scaling and correlation on multitask gain\. We report the relative ROC AUC improvement \(Δm%\\Delta\_\{m\}\\%\) of TabPFN\-MT compared to a single\-task baseline\. As the model divides its fixed capacity across more targets, the multitask performance advantage gradually decreases\. This drop\-off is most severe for datasets with low inter\-task correlation \(correlation=0\.2\\text\{correlation\}=0\.2\), indicating an information bottleneck when modelling weakly related targets\.To better understand the capacity limitations of joint inference, we ablate the effect of increasing the number of targets on predictive performance by sweeping the task correlation from 0\.2 to 0\.8\. In[Figure˜6](https://arxiv.org/html/2605.20234#A9.F6)we see that the model’s performance relative to a single\-task baseline tends to gradually decrease as the number of tasks increases\. This makes sense as TabPFN\-MT must divide its capacity across modelling each target whereas the baseline uses increasing levels of compute as the number of tasks increases\. We observe that the drop\-off in performance is more severe in cases with lower task correlation\. This does indicate that there is an information bottleneck in the decoder during joint inference which could motivate different approaches to decoding in future work\.

## Appendix JExtended Computational Efficiency Analysis

This section expands upon the computational efficiency findings discussed in[Section˜4\.3](https://arxiv.org/html/2605.20234#S4.SS3)\. To rigorously evaluate the cost of multi\-target scaling across diverse model architectures, we utilize two complementary metrics: a hardware\-agnostic FLOPs estimation for deep learning models \([Section˜J\.1](https://arxiv.org/html/2605.20234#A10.SS1)\), and an end\-to\-end wall\-clock runtime analysis \([Section˜J\.2](https://arxiv.org/html/2605.20234#A10.SS2)\) designed to capture the practical optimization overhead of tree\-based baselines like GBDTs\.

### J\.1FLOPs Estimation Methodology

For a rigourous and hardware\-agnostic comparison of computational cost, we measure the total Floating Point Operations \(FLOPs\) required for the training and inference of each architecture\. We use Pytorch’s native profiler \(`torch\.profiler`\) for all models\.

Our estimation procedure is defined as follows:

1. 1\.Hyperparameter Optimization \(HPO\):For all trainable baselines, we first execute an HPO search via Optuna to determine the optimal, dataset\-specific architecture \(e\.g\., layer depth, embedding dimensions\) and maximum training epochs\. Note that the FLOPs required to conduct this search are omitted from our final figures; our reported FLOPs represent only the cost of the single, final optimal fit, making our comparison deliberately conservative in favor of the baselines\.
2. 2\.Representative Profiling:We isolate a single, representative forward batch and a single complete training step \(forward pass, loss calculation, backward pass, and optimizer step\)\. We profile these steps using ‘torch\.profiler‘, recording the median FLOPs across multiple trials to ensure stability\.
3. 3\.Extrapolation:The profiled training step FLOPs are multiplied by the total number of batches per epoch and the true number of executed epochs \(accounting for early stopping\)\. Inference FLOPs are similarly extrapolated by scaling the forward\-batch FLOPs by the total required inference batches\.
4. 4\.multitask Accounting:For single\-target baselines applied to multi\-target datasets, independent models are instantiated, tuned, and evaluated for each of theTTtargets\. Their individual computational footprints are summed, explicitly demonstrating the𝒪​\(T\)\\mathcal\{O\}\(T\)scaling penalty\. Native multitask models process all targets jointly, requiring only a single training and inference cycle \(𝒪​\(1\)\\mathcal\{O\}\(1\)\)\.

### J\.2Runtime Analysis

While FLOPs are a hardware\-agnostic metric, they do not capture the performance of tree\-based models like GBDTs, which rely on logical splits rather than matrix multiplications\. To capture end\-to\-end pipeline efficiency, we conduct a supplementary wall\-clock runtime analysis\.

All runtime benchmarking experiments were executed on a single NVIDIA RTX 5000 GPU utilizing 16 CPU threads\. To mitigate timing variance from hardware initialization and data loading overhead, each evaluation is repeated three times, and the median runtime is reported\.

As illustrated in[Figure˜7](https://arxiv.org/html/2605.20234#A10.F7), single\-target baselines incur a linear wall\-clock penalty \(𝒪​\(T\)\\mathcal\{O\}\(T\)\) as the target space grows\. When scaling from 1 to 5 targets, single\-task methods \(including GBDTs\) have a scaling factor around5\.0×5\.0\\times, because they require independent HPO, fitting, and inference for every variable\. multitask deep learning baselines process targets jointly, but still incur optimization overhead that scales with the complexity of the joint loss landscape and the necessary HPO phase\.

Because TabPFN\-MT requires absolutely no dataset\-specific gradient updates or HPO search, its wall\-clock scaling factor remains near1\.0×1\.0\\times\(𝒪​\(1\)\\mathcal\{O\}\(1\)\) regardless of target count\. A detailed breakdown of these computational costs, including the exact wall\-clock penalties introduced by the HPO phase across varying target counts, is provided in[Table˜11](https://arxiv.org/html/2605.20234#A10.T11)\.

![Refer to caption](https://arxiv.org/html/2605.20234v1/cost_of_targets_scaling_log.png)Figure 7:Cost of target scaling across architectures\. Total runtime \(fit and inference\) is evaluated on synthetic datasets containing 1 to 5 targets\. While single\-target baselines exhibit linear scaling penalties \(𝒪​\(T\)\\mathcal\{O\}\(T\)\), our proposed multitask architecture maintains near\-constant execution time \(𝒪​\(1\)\\mathcal\{O\}\(1\)\), significantly outperforming both single\- and multi\-target baselines\.Table 11:Median per\-dataset timing comparison for T=1 vs T=5 with scaling\.