Training Fair Tabular Foundation Models

# Training Fair Tabular Foundation Models
Source: [https://arxiv.org/html/2608.14211](https://arxiv.org/html/2608.14211)
Patrik KenfackAffiliation:ÉTS MontréalAffiliation:Mila \- Quebec AI InstituteEmail:[patrik\-joslin\.kenfack\.1@ens\.etsmtl\.ca](mailto:)Anthony L\. CateriniAffiliation:Layer 6 AIAffiliation:Toronto, CanadaEmail:[anthony@layer6\.ai](mailto:)Samira Ebrahimi KahouAffiliation:University of CalgaryAffiliation:Mila \- Quebec AI Institute, CIFAREmail:[samira\.ebrahimikahou@ucalgary\.ca](mailto:)Ulrich AïvodjiAffiliation:ÉTS MontréalAffiliation:Mila \- Quebec AI InstituteEmail:[ulrich\.aivodji@etsmtl\.ca](mailto:)

###### Abstract

Tabular Foundation Models \(TFMs\) have emerged as leading methods for tabular predictive tasks, leveraging in\-context learning to predict on new data without task\-specific training\. Despite the increased use of TFMs in high\-stakes decision\-making, their fairness properties remain largely unexplored\. In this work, we incorporate fairness constraints directly into TFM training, enabling fair predictions in a single forward pass\. Our approach addresses two key challenges: limited access to sensitive attributes in training data, and the incompatibility of existing fairness techniques with the in\-context learning paradigm\. We proposeFairTFM, a scalable training strategy based on synthetic fairness tasks and a fairness\-aware architecture using a gradient reversal layer, which encourages the model to learn representations invariant to sensitive attributes\. Experiments on 132 fairness tasks show consistent improvements in fairness while maintaining competitive accuracy\.

## 1Introduction

Tabular data, organized in rows and columns, is the dominant modality for decision\-making tasks in domains such as healthcare and finance\. While tree\-based models, such as XGBoost\[[8](https://arxiv.org/html/2608.14211#bib.bib18)\], have long dominated tabular learning, tabular foundation models \(TFMs\) have emerged as strong alternatives\[[21](https://arxiv.org/html/2608.14211#bib.bib34)\]\. These models are typically pretrained on large collections of tabular data and can adapt to new tasks via in\-context learning \(ICL\)\[[7](https://arxiv.org/html/2608.14211#bib.bib9)\], using only a few labeled examples\. Unlike traditional approaches, TFMs do not require task\-specific training or hyperparameter tuning\. Notably, models such as TabDPT\[[29](https://arxiv.org/html/2608.14211#bib.bib11)\], TabPFNv2\[[22](https://arxiv.org/html/2608.14211#bib.bib5)\], and TabICLv2\[[35](https://arxiv.org/html/2608.14211#bib.bib1)\]can match or outperform heavily tuned tree\-based models across a wide range of datasets\[[14](https://arxiv.org/html/2608.14211#bib.bib12)\]\.

Even as TFMs are deployed in high\-stakes decision\-making, their fairness properties remain largely unexplored\. Recent work shows that, despite strong predictive performance, TFMs can exhibit biased outcomes similar to traditional models\[[24](https://arxiv.org/html/2608.14211#bib.bib6)\]\. While fairness\-aware training has been extensively studied in classical machine learning\[[31](https://arxiv.org/html/2608.14211#bib.bib16)\], these approaches do not readily extend to the ICL paradigm, where predictions must be produced in a single forward pass without task\-specific optimization\. This gap calls for fairness\-aware training methods specifically designed for TFMs\. Hence, in this work, we propose a training framework that incorporates fairness as a first\-class objective in TFMs alongside predictive performance\. We train TFMs under explicit fairness constraints, enabling fair predictions directly through ICL without requiring post\-hoc correction or task\-specific retraining\. To our knowledge, the resulting model is the first TFM that accounts for statistical group fairness notions, such as demographic parity and equalized odds, during pretraining\.

As a running example, consider a bank that uses a TFM to predict whether a client’s income will exceed $50,000\. Anti\-discrimination regulations forbid disadvantaging clients based on gender, which could occur downstream if the income predictions are biased\. Since an off\-the\-shelf TFM makes predictions without modification to its weights, there is no opportunity to apply fairness constraints specific to this task\. With our approach, the bank simply marks gender as the sensitive attribute in the context set, and the pretrained model produces fairer predictions in a single forward pass\.

Our method for fair pretraining relies on two key components\. \(i\)Synthetic fairness task generation:given a dataset, we randomly designate an input feature as a sensitive attribute, treat it as categorical, and optimize for fairness with respect to it\. Repeating this process for every dataset sampled during pretraining enables scalable fairness\-aware training across diverse tasks\. \(ii\)Fairness\-aware architecture and training:we extend transformer\-based TFMs with a dedicated encoder for sensitive attributes and introduce a dual\-head prediction mechanism\. In addition to the label prediction head, we include a sensitive attribute predictor connected through a gradient reversal layer\[[16](https://arxiv.org/html/2608.14211#bib.bib20)\], encouraging the model to learn representations invariant to sensitive attributes\.

We evaluate our approach on 120 fairness tasks derived from the ACS PUMS datasets\[[12](https://arxiv.org/html/2608.14211#bib.bib21)\]with varying sensitive attributes, and on 12 additional tasks built from six widely used fairness benchmarks to test generalization\. We measure fairness using demographic parity, equal opportunity, and equalized odds, comparing against strong baselines\. Our results show that the proposedFairTFMframework consistently improves fairness metrics while maintaining competitive predictive performance, improving fairness by 32–75% relative to the strongest TFM baseline at accuracy costs of 2–11%\.

Our contributions are as follows:

- •We propose a scalable strategy for creating fairness tasks with a wide variety of sensitive attribute relationships, based on a stick\-breaking group construction that requires no labelled sensitive attributes\.
- •We design a fairness\-aware transformer architecture using adversarial learning that enables fair predictions in a single forward pass, handling a sensitive attribute that changes identity across tasks and is masked at inference\.
- •We provide an extensive empirical evaluation on a diverse set of tasks, demonstrating improved fairness metrics while preserving accuracy, including comparisons with fairness\-aware baselines and probing analyses of the learned representations\.

## 2Related work

FairnessPrior work on fair machine learning typically intervenes at one of three stages: pre\-processing the data, incorporating fairness constraints during training, or post\-processing model outputs\[[23](https://arxiv.org/html/2608.14211#bib.bib25),[42](https://arxiv.org/html/2608.14211#bib.bib24),[20](https://arxiv.org/html/2608.14211#bib.bib23),[43](https://arxiv.org/html/2608.14211#bib.bib27)\]\. These approaches have been studied extensively for conventional supervised models, but they often assume access to the training pipeline or to calibrated model outputs\. This assumption is less compatible with ICL, where a pretrained model is used as a frozen predictor at inference time\. Our setting is therefore closer to learning fairness\-aware representations during pretraining so that fair behaviour can be obtained in a single forward pass\. While adversarial fairness objectives based on gradient reversal have previously been studied in unsupervised domain adaptation\[[15](https://arxiv.org/html/2608.14211#bib.bib13)\], integrating such objectives into TFM pretraining introduces fundamentally different challenges, including heterogeneous tasks, dynamically changing sensitive attributes across sampled tasks, and the requirement of single\-pass in\-context inference without task\-specific optimization\. Our contribution is therefore not merely the use of adversarial learning for fairness, but the introduction of a scalable fairness\-aware pretraining framework for TFMs that enables fairer predictions across unseen downstream tasks while preserving the inference properties of in\-context learning\.

Tabular foundation modelsRecent tabular foundation models such as TabPFN, TabDPT, and TabICL demonstrate that pretrained transformers can be highly competitive on tabular prediction tasks\[[21](https://arxiv.org/html/2608.14211#bib.bib34),[22](https://arxiv.org/html/2608.14211#bib.bib5),[29](https://arxiv.org/html/2608.14211#bib.bib11),[19](https://arxiv.org/html/2608.14211#bib.bib4),[34](https://arxiv.org/html/2608.14211#bib.bib2),[35](https://arxiv.org/html/2608.14211#bib.bib1)\]\. However, this literature emphasizes predictive accuracy, and does not directly address the potential for biased outcomes\. FairPFN\[[36](https://arxiv.org/html/2608.14211#bib.bib17)\]does incorporate fairness into TFM pretraining under a causal notion of fairness, which contrasts with our work which targets statistical group fairness notions such as demographic parity, equal opportunity, and equalized odds\. Statistical fairness is data\-driven and focuses on distributional fairness \(outcome\-based\), while causal fairness is more interventional and does not necessarily ensure outcome parity across groups\. Our work focuses directly on optimizing statistical fairness notions commonly used in tabular classification benchmarks\[[27](https://arxiv.org/html/2608.14211#bib.bib19)\]\. We include FairPFN as a baseline in our experiments and find that its causal objective does yield partial group\-fairness benefits, but at a predictive cost that leaves it Pareto\-dominated byFairTFM, and without any mechanism for controlling the fairness–utility trade\-off\. We provide a more detailed discussion of these connections in Appendix[A](https://arxiv.org/html/2608.14211#A1)\.

## 3Training fair tabular foundation models

In this section, we present our fairness\-aware tabular foundation model for statistical fairness\. We describe how fairness tasks are synthesized from generic data priors, introduce the model architecture, and specify the pretraining objective\.

![Refer to caption](https://arxiv.org/html/2608.14211v1/figure_step_1.png)\(a\)Fairness task sampling from a prior\.
\(b\)Overview of theFairTFMarchitecture\.
Figure 1:\(a\) We construct fairness tasks from a prior data generator by first sampling a dataset\(X′,y\)\(X^\{\\prime\},y\), then randomly removing one feature to serve as the sensitive attributessand converting it to a categorical variable, yielding the triplet\(X,y,s\)\(X,y,s\)\. The resulting data are split into context\(Xctx,yctx,sctx\)\(X\_\{\\text\{ctx\}\},y\_\{\\text\{ctx\}\},s\_\{\\text\{ctx\}\}\)and query\(Xqy,yqy,sqy\)\(X\_\{\\text\{qy\}\},y\_\{\\text\{qy\}\},s\_\{\\text\{qy\}\}\)sets for ICL\. \(b\)FairTFMjointly processes context and query inputs with a transformer\. In addition to the main prediction head for the target labely^qy\\hat\{y\}\_\{\\text\{qy\}\}, a second head predicts the sensitive attributes^qy\\hat\{s\}\_\{\\text\{qy\}\}\. A gradient reversal layer on the sensitive branch \(red arrow\) encourages the model to learn representations that are invariant toss, thereby reducing reliance on sensitive\-correlated features while preserving predictive performance\.### 3\.1Fairness task sampling from a dataset prior

Most TFMs are pretrained on tasks sampled from a dataset prior, whether synthetic or representing real datasets, in order to expose the model to a broad distribution of supervised learning problems\. We follow the same principle, but augment it to generate*fairness tasks*in a self\-supervised manner\[[9](https://arxiv.org/html/2608.14211#bib.bib14),[4](https://arxiv.org/html/2608.14211#bib.bib15)\]that captures diverse forms of group\-dependent bias\. A similar process was employed in TabDPT\[[29](https://arxiv.org/html/2608.14211#bib.bib11)\]for crafting diverse predictive tasks from real\-world datasets, and in CausalPFN\[[3](https://arxiv.org/html/2608.14211#bib.bib10),[37](https://arxiv.org/html/2608.14211#bib.bib3)\]for generating synthetic causal inference tasks\.

Concretely, starting from a sampled dataset, we randomly designate one input feature as the sensitive attribute and remove it from the feature set, as illustrated in Figure[1\(a\)](https://arxiv.org/html/2608.14211#S3.F1.sf1)\. When the selected feature is continuous, we convert it into a categorical attribute using a “stick\-breaking” discretization scheme\. We first sample a sequence of proportions from Beta distributions and use them to construct mixture weights via a Dirichlet process\[[25](https://arxiv.org/html/2608.14211#bib.bib31)\], where each component represents a fraction of the remaining mass\. These weights are then mapped to empirical quantiles of the normalized feature, producing data\-dependent cut points that partition the feature into discrete groups\. This procedure yields flexible, non\-uniform bins that adapt to the underlying feature distribution while retaining a probabilistic interpretation\.

In the running example, the bank’s dataset corresponds to one predictive task requiring fairness considerations, where gender plays the role of the designated sensitive attribute\. During pretraining, any feature in a dataset can play the role of sensitive attribute in some sampled tasks, and an ordinary predictive feature in others\.

### 3\.2Network architecture

We build on TabPFN’s transformer encoder for tabular data\[[40](https://arxiv.org/html/2608.14211#bib.bib32)\], using the nanoTabPFN architecture\[[33](https://arxiv.org/html/2608.14211#bib.bib33)\]as a lightweight backbone that also has less input pre\-processing so that we mitigate confounding effects in our controlled experiments\. In this architecture, each input pair\(Xi,yi\)\(X\_\{i\},y\_\{i\}\)is represented as a sequence ofdd\-dimensional tokens and is processed by alternating self\-attention over rows and columns, enabling ICL along both axes\.

During pretraining, available input data is divided into context\(Xctx,yctx\)\(X\_\{\\mathrm\{ctx\}\},y\_\{\\mathrm\{ctx\}\}\)and query\(Xqy,yqy\)\(X\_\{\\mathrm\{qy\}\},y\_\{\\mathrm\{qy\}\}\)sets\. As in TabPFN, the query labelsyqyy\_\{\\mathrm\{qy\}\}are masked and the corresponding output token is mapped to class logits through a multi\-layer perceptron \(MLP\)\. For inference, available labelled data \(what would be training data for a non\-foundational predictive model\) is provided as context, and the TFM predictsyqyy\_\{\\mathrm\{qy\}\}on unlabelled query data\.

Our fairness\-aware extension, which we call Fair Tabular Foundation Model \(FairTFM\), is shown in Figure[1\(b\)](https://arxiv.org/html/2608.14211#S3.F1.sf2)\. Rather than operating only on pairs\(xi,yi\)\(x\_\{i\},y\_\{i\}\),FairTFMprocesses triplets\(xi,yi,si\)\(x\_\{i\},y\_\{i\},s\_\{i\}\), wheresis\_\{i\}denotes thedd\-dimensional token of the sensitive attribute\.FairTFMincorporates a shared transformer backbone that learns representations, and two predictive MLP heads for the target labely^qy\\hat\{y\}\_\{\\text\{qy\}\}and sensitive attributes^qy\\hat\{s\}\_\{\\text\{qy\}\}\. Because the chosen attention mechanism uses no column positional encodings, predictions are invariant to the ordering of input columns\[[33](https://arxiv.org/html/2608.14211#bib.bib33)\]\. Hence, when a feature is randomly designated as the sensitive attribute during pretraining, its index in the table does not have an impact on predictions\.

To discourage the learned representation from encoding sensitive information, the secondary prediction head that predictss^qy\\hat\{s\}\_\{\\text\{qy\}\}is preceded by a gradient reversal layer \(GRL\)\[[15](https://arxiv.org/html/2608.14211#bib.bib13)\]\. The head itself is trained to minimize prediction error ofsqys\_\{\\text\{qy\}\}, and would naturally exploit any information aboutssencoded in the transformer’s representation\. However, the GRL multiplies the gradient signal passing to the transformer by−1\-1, such that the representation is adversarially trained to contain no information aboutss, making the prediction head’s task harder\. Intuitively, the GRL forces the transformer to reduce reliance on rows and features that are informative forss, including proxies forss, leading to fairer predictions when the same representation is used by theyyhead\.

A key advantage of the GRL formulation in our setting is that it does not require training a separately optimized adversary for each sampled fairness task, even though the designated sensitive attribute changes across tasks during pretraining\. In contrast, an adversarial setup with separately trained classifier adversaries would require fitting or re\-optimizing a new adversary for each sampled fairness task during pretraining\. Such per\-task adversarial training would substantially increase pretraining complexity and make large\-scale task sampling less practical\.

Since the fairness tasks used for pretraining sample the sensitive attribute at random from the available features \(Section[3\.1](https://arxiv.org/html/2608.14211#S3.SS1)\), one may worry that the GRL degrades the embedding broadly rather than removing only sensitive information\. The key design point is that the sensitive attribute is removed fromX′X^\{\\prime\}and provided through a dedicated encoder as an explicit token for every context row \(Algorithm[1](https://arxiv.org/html/2608.14211#alg1), lines 3 and 8–9\)\. Thess\-head only predictsss, so the reversed gradient only penalizes information about the attribute signalled by the in\-contextsstokens; the learned invariance tossis conditional and in\-context\. Moreover, since each feature ofX′X^\{\\prime\}is treated as sensitive in some sampled tasks and not in others, the overall objective is not optimized by degrading any fixed feature\. In the running example, the query representations suppress gender*and its proxies*because the bank designates gender as the sensitive attribute in the context examples\.

At inference time, we do not assume access to the sensitive attributesqys\_\{\\mathrm\{qy\}\}\. Since the transformer is trained to create representations that do not contain information aboutsqys\_\{\\mathrm\{qy\}\}, providing it would not be particularly fruitful\. Hence, we replacesqys\_\{\\mathrm\{qy\}\}values with a fixed learnable parameter during both pretraining and inference\. This allows the transformer encoder to operate ondd\-dimensional tokens representing\(xi,yi,si\)\(x\_\{i\},y\_\{i\},s\_\{i\}\)and produce query sample embeddings, while ensuring thatsqys\_\{\\mathrm\{qy\}\}is never directly exposed to the model\. Beyond obviating the need for direct access to sensitive information at inference, this design discourages the model from exploitingsqys\_\{\\mathrm\{qy\}\}as a predictive shortcut, which would undermine the objective of the GRL\.

### 3\.3Training objective

Algorithm 1FairTFMpretraining1:Data prior

p⁡\(𝒟\)p\(\\mathcal\{D\}\),FairTFMparameters

θ\\theta, fairness weight

λ\\lambda
2:whilenot convergeddo

3:Sample a dataset

\(X′,y\)∼p⁡\(𝒟\)\(X^\{\\prime\},y\)\\sim p\(\\mathcal\{D\}\)
4:Produce

XXby randomly removing one feature from

X′X^\{\\prime\}as the sensitive attribute

ss
5:if

ssis continuousthen

6:Discretize

ssinto categorical groups via “stick\-breaking” quantile bins

7:endif

8:Split

\(X,y,s\)\(X,y,s\)into context

\(Xctx,yctx,sctx\)\(X\_\{\\mathrm\{ctx\}\},y\_\{\\mathrm\{ctx\}\},s\_\{\\mathrm\{ctx\}\}\)and query

\(Xqy,yqy,sqy\)\(X\_\{\\mathrm\{qy\}\},y\_\{\\mathrm\{qy\}\},s\_\{\\mathrm\{qy\}\}\)
9:Mask

yqyy\_\{\\mathrm\{qy\}\}and replace

sqys\_\{\\mathrm\{qy\}\}with the learnable sensitive attribute parameter

10:Encode

XiX\_\{i\},

yiy\_\{i\}, and

sis\_\{i\}into

dd\-dimensional vectors using their respective MLP encoders

11:Encode the

dd\-dimensional triplets

\(Xi,yi,si\)\(X\_\{i\},y\_\{i\},s\_\{i\}\)with the shared transformer encoder

12:Predict query labels

y^qy\\hat\{y\}\_\{\\mathrm\{qy\}\}with the main head

13:Predict query sensitive attributes

s^qy\\hat\{s\}\_\{\\mathrm\{qy\}\}with the adversarial head through a GRL

14:Compute

ℒ=CE⁡\(y^qy,yqy\)\+λ​CE​\(s^qy,sqy\)\\mathcal\{L\}=\\mathrm\{CE\}\(\\hat\{y\}\_\{\\mathrm\{qy\}\},y\_\{\\mathrm\{qy\}\}\)\+\\lambda\\,\\mathrm\{CE\}\(\\hat\{s\}\_\{\\mathrm\{qy\}\},s\_\{\\mathrm\{qy\}\}\)
15:Update

θ\\thetaby backpropagation; the GRL reverses gradients from the sensitive\-attribute head into the encoder

16:endwhile

FairTFMis trained from scratch on purely synthetic fairness tasks derived from TabICL’s prior\[[34](https://arxiv.org/html/2608.14211#bib.bib2)\]\(Appendix[D](https://arxiv.org/html/2608.14211#A4)\)\. The transformer, shown in Figure[1\(b\)](https://arxiv.org/html/2608.14211#S3.F1.sf2), produces an embedding that captures row\- and feature\-wise interactions among the context instances, and the two MLP heads output logits for the query label \(y^qy\\hat\{y\}\_\{\\text\{qy\}\}\) and corresponding sensitive attribute \(s^qy\\hat\{s\}\_\{\\text\{qy\}\}\)\. We optimize the model using the following joint cross\-entropy objective:

ℒ=CE​\(y^qy,yqy\)\+λ⋅CE​\(s^qy,sqy\)\.\\mathcal\{L\}=\\text\{CE\}\(\\hat\{y\}\_\{\\text\{qy\}\},y\_\{\\text\{qy\}\}\)\+\\lambda\\cdot\\text\{CE\}\(\\hat\{s\}\_\{\\text\{qy\}\},s\_\{\\text\{qy\}\}\)\.\(1\)
Minimizingℒ\\mathcal\{L\}encourages accurate prediction of both the target labelyyand the sensitive attributess, while the GRL before thesshead ensures the encoder learns features that are not predictive ofss\. This objective naturally induces a fairness\-accuracy trade\-off, especially when the target labelyyand sensitive attributessare correlated\. The parameterλ\\lambdain Eq\.[1](https://arxiv.org/html/2608.14211#S3.E1)controls the strength of this trade\-off\. As summarized in Algorithm[1](https://arxiv.org/html/2608.14211#alg1), each pretraining step instantiates a new fairness task, forms context and query sets for ICL, masks the query label and sensitive attribute to preserve the intended inference setting, and then updates the shared transformer encoder and other components through the coupled label\-prediction and sensitive\-attribute losses\.

## 4Results

In this section, we describe the experimental setup, including the evaluation tasks, baseline models, and metrics, before discussing the main empirical results\. To facilitate reproducibility, we provide the inference code, pretrained checkpoints, and scripts for reproducing the main results at[github\.com/patrikken/FairTFM\-inference](https://github.com/patrikken/FairTFM-inference)\.

### 4\.1Experimental setup

DatasetsWe evaluate our model on 120 fairness tasks derived from the 2018 1\-Year American Community Surveys\[[12](https://arxiv.org/html/2608.14211#bib.bib21)\], accessed through thefolktableslibrary, which is released under the MIT License\.111https://github\.com/socialfoundations/folktablesWe generate these tasks by varying three factors: \(i\) the prediction problem \(ACSIncome, ACSPublicCoverage, ACSMobility, ACSEmployment, and ACSTravelTime\), \(ii\) the sensitive attribute \(Gender, Age, and Race\), and \(iii\) the state from which the data is sampled \(eight US states\)\. This yields a total of5×3×8=1205\\times 3\\times 8=120real\-world fairness tasks not used during training\. Additional details are provided in Appendix[B](https://arxiv.org/html/2608.14211#A2)\. To test generalization beyond a single data source, we further evaluate on 12 fairness tasks built from six widely used non\-ACS benchmarks \(Appendix[E\.6](https://arxiv.org/html/2608.14211#A5.SS6)\)\.

Base modelsWe compare our method against standard machine learning baselines, including logistic regression \(LR\), random forest \(RF\), XGBoost \(XGB\), andkk\-nearest neighbours \(KNN\) in their default scikit\-learn configuration\[[32](https://arxiv.org/html/2608.14211#bib.bib43)\]\. We also include recent TFMs, namely TabPFNv2\.5 and TabICLv2\. Because these models do not incorporate any fairness intervention, they provide strong baselines for assessing the fairness limitations of existing approaches on our benchmark\. These models do not receivesqys\_\{\\text\{qy\}\}at inference, similar toFairTFM\. We additionally compare against FairPFN\[[36](https://arxiv.org/html/2608.14211#bib.bib17)\], the only prior TFM that incorporates fairness during pretraining\. FairPFN targets counterfactual fairness rather than the statistical notions we optimize, so it is not designed to minimize DP, EOD, or EOP; we include it as the closest existing approach to ours, to test whether pretraining for a causal fairness notion also yields group\-fairness benefits\. We use the publicly released FairPFN checkpoint, so its backbone size and pretraining prior differ from ours; this comparison should therefore be read as a comparison between published methods rather than as a controlled ablation of the fairness objective\.

MetricsIn addition to accuracy and AUCROC, we report three fairness metrics: demographic parity difference \(DP\), equalized odds difference \(EOD\), and equal opportunity difference \(EOP\)\. Definitions and implementation details are provided in Appendix[C](https://arxiv.org/html/2608.14211#A3)\. Unless otherwise noted, we report metric values averaged across all 120 tasks to summarize overall fairness–accuracy trends\.

### 4\.2Experimental Results

In this section, we present a comprehensive empirical evaluation ofFairTFMacross three complementary settings\. First, we benchmark its performance in terms of both accuracy and fairness against classical methods and recent tabular foundation models\. Second, we compare against fairness\-aware variants of classical approaches to assess trade\-offs under explicit fairness constraints\. Third, we evaluateFairTFMas a fairness\-enhancing representation learner, examining its ability to produce embeddings that support fairer downstream predictions\.

Figure 2:Pareto front between accuracy and fairness for various models\.↑\\uparrowindicates higher is better \(accuracy\) and↓\\downarrowindicates smaller is better \(unfairness\)\.Fairness\-accuracy trade\-off across tasksFigure[2](https://arxiv.org/html/2608.14211#S4.F2)shows our main results: the Pareto front between accuracy and fairness for various models\. ForFairTFM, we pretrain variants withλ∈\{0\.1,0\.7,10,25\}\\lambda\\in\\\{0\.1,0\.7,10,25\\\}to control the fairness–accuracy trade\-off, and display that trade\-off for several pretraining checkpoints to more fully trace out the frontier\. TabICLv2 and TabPFNv2\.5 achieve the highest average accuracy across tasks, but they also occupy the most unfair region of the Pareto frontier, suggesting that their predictive gains come with a substantial fairness cost\. In contrast, the classical machine learning baselines are generally less accurate, yet they remain similarly unfair\.

FairTFMconsistently improves fairness metrics, and for a range ofλ\\lambdaparameters maintains competitive—and often stronger—accuracy compared to classical baselines such as LR and KNN\. Table[2](https://arxiv.org/html/2608.14211#A5.T2)in Appendix[E\.1](https://arxiv.org/html/2608.14211#A5.SS1)complements Figure[2](https://arxiv.org/html/2608.14211#S4.F2)by reporting the corresponding aggregated metrics over all 120 tasks, including standard deviations across three random seeds\. Whereas Figure[2](https://arxiv.org/html/2608.14211#S4.F2)emphasizes the Pareto frontier traced out by different checkpoints and fairness weights, Table[2](https://arxiv.org/html/2608.14211#A5.T2)makes the overall pattern explicit at the level of final checkpoints: the strongest unconstrained TFMs achieve the highest accuracy but are also the least fair, whileFairTFMvariants deliver substantial reductions in DP, EOD, and EOP with only a moderate loss in predictive performance\. Table[3](https://arxiv.org/html/2608.14211#A5.T3)in Appendix[E\.2](https://arxiv.org/html/2608.14211#A5.SS2)reports the relative changes against the strongest TFM \(TabPFNv2\.5\), the strongest classical model \(XGB\), and LR:FairTFMimproves fairness by 32–75% for accuracy costs of 2–11%, and even dominates LR on all metrics atλ=0\.7\\lambda=0\.7\. Tables[2](https://arxiv.org/html/2608.14211#A5.T2)and[3](https://arxiv.org/html/2608.14211#A5.T3)report the last pretraining checkpoint of eachFairTFMvariant, while the Pareto\-dominant checkpoints visible in Figures[2](https://arxiv.org/html/2608.14211#S4.F2)and[3](https://arxiv.org/html/2608.14211#S4.F3)yield better trade\-offs, but would require model selection on a validation set\.

The results also highlight the role ofλ\\lambdain shaping the accuracy\-fairness trade\-off\. Larger values ofλ\\lambda\(e\.g\.,λ=25\\lambda=25\) move the model toward the fairer region of the Pareto frontier at the expense of predictive accuracy\. Smaller values ofλ\\lambda, by contrast, prioritize accuracy and therefore place the model in less fair regions of the frontier\. The Pareto front ends up tracing a nearly linear path through accuracy\-unfairness space, withFairTFMfilling out a large segment\. These trends indicate that our training objective provides a simple and effective mechanism for navigating different operating points depending on the fairness requirements of the application\. We observe the same trend when measuring predictive performance using AUCROC, as shown in Figure[7](https://arxiv.org/html/2608.14211#A5.F7)of Appendix[E](https://arxiv.org/html/2608.14211#A5)\.

FairPFN behaves differently from the unconstrained baselines: it is fairer than all of them on all three metrics, and in Table[2](https://arxiv.org/html/2608.14211#A5.T2)attains the lowest DP of any final checkpoint \(0\.043\)\. Pretraining for a causal fairness notion therefore does transfer part of its benefit to statistical group fairness, supporting our broader claim that fairness can be acquired during pretraining and translate to ICL predictions\. However, for FairPFN this comes at a large predictive cost; FairPFN has the lowest AUCROC of all models we evaluate here \(0\.695\)\. In every panel of Figures[2](https://arxiv.org/html/2608.14211#S4.F2)and[7](https://arxiv.org/html/2608.14211#A5.F7)FairTFMprovides operating points that Pareto\-dominate it, being simultaneously fairer and more predictive\. Just as important, FairPFN exposes no mechanism for navigating the fairness–utility trade\-off: it yields a single operating point, whereasλ\\lambdalets a practitioner choose one that matches the application\. Part of the predictive gap is expected, since FairPFN optimizes fair outcomes under interventions on structural causal models rather than accuracy on real labels; the comparison is thus evidence that the two fairness notions are not interchangeable, not that FairPFN fails at its own objective\.

Figure 3:Pareto front between accuracy and fairness for various models on the 12 fairness tasks beyond ACS PUMS \(Appendix[E\.6](https://arxiv.org/html/2608.14211#A5.SS6)\)\.↑\\uparrowindicates higher is better \(accuracy\) and↓\\downarrowindicates smaller is better \(unfairness\)\.Generalization beyond ACS PUMS tasksFigure[3](https://arxiv.org/html/2608.14211#S4.F3)shows the Pareto front on the 12 non\-ACS tasks\. The conclusions match those from Figure[2](https://arxiv.org/html/2608.14211#S4.F2): TabICLv2 and TabPFNv2\.5 are the most accurate but occupy the most unfair region, whileFairTFMvariants trace the low\-unfairness end of the front, withλ\\lambdacontrolling the trade\-off\. The AUCROC view \(Figure[13](https://arxiv.org/html/2608.14211#A5.F13)in Appendix[E\.6](https://arxiv.org/html/2608.14211#A5.SS6)\) further shows thatFairTFMimproves fairness without a commensurate loss in ranking performance\. FairPFN behaves as on the ACS benchmark: it is fairer than every unconstrained baseline on all three metrics, but at a predictive cost that again leaves it Pareto\-dominated byFairTFM, under both accuracy and AUCROC\. Table[4](https://arxiv.org/html/2608.14211#A5.T4)in Appendix[E\.6](https://arxiv.org/html/2608.14211#A5.SS6)reports the corresponding aggregated metrics and demonstrates the gap directly\. At essentially the same accuracy \(0\.734 vs\. 0\.737\),FairTFM\-10 is fairer than FairPFN on all three metrics and 7\.5 points higher in AUCROC\.

Figure 4:Training dynamics comparison between nanoTabPFN \(without fairness constraint\) andFairTFMwithλ∈\{0\.7,25\}\\lambda\\in\\\{0\.7,25\\\}\.Performance dynamics during trainingFigure[4](https://arxiv.org/html/2608.14211#S4.F4)provides a finer\-grained view of how fairness emerges over the course of pretraining, tracking accuracy and fairness \(DP and EOD\) forFairTFMwithλ∈\{0\.7,25\}\\lambda\\in\\\{0\.7,25\\\}and for nanoTabPFN across checkpoints\. BothFairTFMvariants achieve a fairness advantage over nanoTabPFN very early in training and retain this advantage throughout optimization\. Since both models share the same data prior, backbone, and training budget, differing only in the fairness\-specific components, this comparison isolates the effect of our intervention \(full discussion in Appendix[E\.3](https://arxiv.org/html/2608.14211#A5.SS3)\)\.

ComparingFairTFMagainst classical fairness\-aware modelsThe results above show thatFairTFMimproves fairness substantially while remaining competitive with standard classical baselines such as LR and KNN\. A stronger comparison, however, is against classical methods that are themselves explicitly optimized for fairness\. To this end, we evaluate three tabular models—LR, RF, and XGB—augmented with the Exponentiated Gradient \(EG\) reduction of[1](https://arxiv.org/html/2608.14211#bib.bib26), which enforces group\-fairness constraints during training\. We focus on ACSIncome in Alabama \(AL\) and instantiate three fairness tasks by varying the sensitive attribute over gender, race, and age\. For each task, we use a random 80/20 train–test split and sweep the EG fairness\-violation tolerance over\[0\.01,0\.02,…,0\.1,0\.2,…,1\.0\]\[0\.01,0\.02,\\ldots,0\.1,0\.2,\\ldots,1\.0\], with finer resolution in the low\-violation regime to better characterize the high\-fairness end of the trade\-off\. We average the results across three random seeds\. This setup allows us to compareFairTFMnot only to strong predictive baselines, but also to established in\-processing fairness methods under a matched downstream evaluation protocol\.

Figure 5:Fairness\-accuracy trade\-off comparison with task\-specific classical models trained with fairness constraints using Exponentiated Gradient \(EG\), on ACSIncome \(Alabama\) with gender as the sensitive attribute\. Results for race and age are shown in Figure[11](https://arxiv.org/html/2608.14211#A5.F11)of Appendix[E](https://arxiv.org/html/2608.14211#A5)\.Figure[5](https://arxiv.org/html/2608.14211#S4.F5)shows thatFairTFMremains highly competitive even against these fairness\-constrained baselines\. Across all three sensitive attributes, its checkpoints trace out a broad Pareto frontier, indicating that a single pretrained model can realize multiple fairness–accuracy operating points without retraining\. For gender,FairTFMattains especially strong trade\-offs for EOD and EOP, matching or improving upon the frontier formed by EG\-based baselines while remaining competitive on DP\. For race \(Figure[11](https://arxiv.org/html/2608.14211#A5.F11)in Appendix[E](https://arxiv.org/html/2608.14211#A5)\), where all methods incur larger fairness gaps,FairTFMstill spans a wide and competitive portion of the frontier, particularly at the lower\-unfairness end\. For age, the comparison is more metric\-dependent: EG\-based classical models achieve stronger accuracy–fairness trade\-offs for EOD and EOP, whereasFairTFMremains competitive on DP but exhibits a clearer trade\-off between predictive performance and these stricter parity criteria\. Overall, the main advantage ofFairTFMis not that it dominates every baseline on every metric, but that it delivers competitive Pareto\-efficient solutions across heterogeneous fairness tasks in a single forward pass, whereas the classical alternatives must be retrained with task\-specific fairness constraints for each new setting\. Moreover, as shown in Figure[12](https://arxiv.org/html/2608.14211#A5.F12)of Appendix[E](https://arxiv.org/html/2608.14211#A5),FairTFMprovides the best overall Pareto front when predictive performance is measured through AUCROC\.

We also evaluated the EG baselines on the 12 non\-ACS tasks with results in Appendix[E\.7](https://arxiv.org/html/2608.14211#A5.SS7)\. EachFairTFMsetting improves the metrics that are relevant to each EG variant—by 20–33% forFairTFM\-0\.7 and 79–82% forFairTFM\-25—while achieving substantially higher AUCROC: EG reduces XGBoost’s AUCROC from 0\.803 to about 0\.70, whereasFairTFM\-0\.7 remains at 0\.803\.

FairTFMas a fair representation learnerWe hypothesize that theFairTFMencoder produces representations that are useful for downstream fair prediction\. Fair representation learning methods learn a mapggof the input that suppresses information about the sensitive attribute while preserving task\-relevant signal; formally:

###### Definition 1\(Fair Representation\)

Let\(X,Y,S\)\(X,Y,S\)be random variables whereX∈ℝm×dX\\in\\mathbb\{R\}^\{m\\times d\}is the input,Y∈𝒴Y\\in\\mathcal\{Y\}is the target label, andS∈𝒮S\\in\\mathcal\{S\}is a sensitive attribute\. Letg:ℝm×d→ℝm×d′g:\\mathbb\{R\}^\{m\\times d\}\\to\\mathbb\{R\}^\{m\\times d^\{\\prime\}\}be a representation map and defineZ:=g⁡\(X\)Z:=g\(X\)\. We say thatZZis a fair representation with respect toSSifZ⟂⟂S\|YZ\\perp\\\!\\\!\\\!\\perp S\\mid Y\.

Definition[1](https://arxiv.org/html/2608.14211#Thmdefinition1)is conceptually aligned with the objective ofFairTFM, which encourages the learned query embedding to suppress sensitive\-attribute information while preserving task\-relevant signal\. We compareFairTFMembeddings against two task\-specific pre\-processing methods from thefairlearnlibrary\[[6](https://arxiv.org/html/2608.14211#bib.bib35)\]: Correlation Remover \(CR\) and Learning Fair Representations \(LFR\)\[[42](https://arxiv.org/html/2608.14211#bib.bib24)\], on ACSIncome \(Alabama\) with gender, race, and age as sensitive attributes\. Notably,FairTFMembeddings are computed in a single forward pass without any task\-specific gradient updates\. For each type of representation as well as the raw features, we train downstream classifiers \(LR, RF, and XGBoost\), and compare their accuracy and fairness properties\. Full protocol details are in Appendix[E\.4](https://arxiv.org/html/2608.14211#A5.SS4)\.

Figure 6:Pareto front of XGBoost models trained with different data representations, with gender as the sensitive attribute\. Results for race and age, and for RF and LR as downstream models, are shown in Appendix[E\.4](https://arxiv.org/html/2608.14211#A5.SS4)\.Figure[6](https://arxiv.org/html/2608.14211#S4.F6)showcases the Pareto front of XGBoost models trained using data representation from different fair methods, with gender as the sensitive attribute\. As can be seen,FairTFMcan provide competitive Pareto points on downstream tasks compared to raw data and dominates LFR, whereas CR severely worsens fairness\. The results show the ability ofFairTFMto generate embeddings that encode less information about the given sensitive attributes while still being predictive of the target\.FairTFMstands out due to its ability to adapt to new tasks via ICL without needing task\-specific optimization\. The remaining sensitive attributes \(race and age\) and downstream models \(RF and LR\) in Appendix[E\.4](https://arxiv.org/html/2608.14211#A5.SS4)show a similar trend\.

Table 1:Linear probe AUC for predicting the sensitive attributessand the labelyyon different representations\. We show AUC mean±\\pmstd over three seeds and the three sensitive attributes on ACSIncome, Alabama\.FairTFMdisplays a controllable trade\-off of reducing the predictability of the sensitive attribute with some obfuscation of the label\.Probe target \(AUC\)Raw inputnanoTabPFNFairTFM\-0\.7FairTFM\-25Sensitive attributess0\.783±\\pm0\.110\.746±\\pm0\.100\.646±\\pm0\.140\.542±\\pm0\.11Labelyy0\.834±\\pm0\.020\.823±\\pm0\.030\.805±\\pm0\.040\.787±\\pm0\.03Probing the learned representationsTo verify that the GRL removes information about the designated sensitive attribute rather than degrading the embedding broadly \(Section[3](https://arxiv.org/html/2608.14211#S3)\), we train linear probes to predict \(a\) the sensitive attributessand \(b\) the labelyyfrom three representations: the raw input featuresXX, the query embeddings of nanoTabPFN, and the query embeddings ofFairTFM\. For the embedding\-based representations, the query embeddings of the initial test set are treated as a new dataset for each probing task\. For both sensitive\-attribute and label prediction, this dataset is randomly split into 80% training and 20% testing subsets\. The probing model \(XGBoost\) is trained on the training split, and performance is evaluated on the held\-out test split using AUC\.

Table[1](https://arxiv.org/html/2608.14211#S4.T1)shows three things\. First, the nanoTabPFN embedding retains most of the sensitive information present in the raw features \(AUC 0\.746 vs\. 0\.783\) even thoughsqys\_\{\\text\{qy\}\}itself is not an input\. Second,FairTFMsubstantially reduces the sensitive\-attribute AUC, approaching random guessing \(0\.542 atλ=25\\lambda=25\)\. Third, task\-relevant information is largely preserved \(AUC 0\.834→\\to0\.805 atλ=0\.7\\lambda=0\.7\)\. These probes support the idea that suppression of sensitive information is attributable to the adversarial objective and is selective\.

## 5Conclusion

We presentFairTFM, the first pretraining framework for tabular foundation models that provides group fairness while preserving the single\-pass inference setting of in\-context learning\. The key idea is to synthesize fairness tasks during pretraining and couple label prediction with adversarial sensitive\-attribute prediction through a gradient reversal layer\. On 120 ACS\-based fairness tasks and 12 additional tasks from six non\-ACS benchmarks, the resulting models consistently improve demographic parity, equal opportunity, and equalized odds while maintaining competitive accuracy\. Beyond the aggregate benchmark results, we showed that these gains emerge throughout training, remain competitive against fairness\-constrained classical baselines, and transfer to downstream fair\-representation settings, with probing experiments confirming that the learned embeddings selectively remove sensitive information\. Taken together, these results suggest that fairness can be incorporated directly into the pretraining of TFMs, rather than introduced only through task\-specific post\-hoc correction as done in prior work\.

#### Limitations and future work

Despite the strong empirical performance ofFairTFM, our study has several limitations\. First, our pretraining strategy relies only on TabICL’s prior generation, which provides scale and diversity but may not capture all of the semantic and societal structure of real sensitive attributes in downstream deployments\. Second, whileFairTFMis broadly competitive, the results also show that it does not dominate specialized fairness\-aware baselines on every metric or every task configuration, especially in the stricter settings where age is used as sensitive attribute\. We view these limitations not as drawbacks of the overall approach, but as evidence that fairness\-aware pretraining opens a rich new research direction\. In particular, our findings pave the way for future work on better fairness\-task priors, improved trade\-off control during pretraining, and foundation models that can adapt their fairness behaviour more explicitly to downstream deployment requirements\. Finally, we emphasize that our experiments rely on nanoTabPFN, a lightweight backbone chosen to isolate the effect of fairness\-aware pretraining\. We believe the proposed framework is largely orthogonal to architectural scaling, and integrating it into larger state\-of\-the\-art TFMs such as TabPFNv2\.5 or TabICLv2 may yield substantially stronger fairness–accuracy trade\-offs through improved representation learning and richer pretrained priors\.

## References

- \[1\]A\. Agarwal, A\. Beygelzimer, M\. Dudik, J\. Langford, and H\. Wallach\(2018\)A reductions approach to fair classification\.InProceedings of the 35th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.80,pp\. 60–69\.Cited by:[§E\.7](https://arxiv.org/html/2608.14211#A5.SS7.p1.1),[§4\.2](https://arxiv.org/html/2608.14211#S4.SS2.p8.1)\.
- \[2\]J\. Angwin, J\. Larson, S\. Mattu, and L\. Kirchner\(2022\)Machine bias\.InEthics of data and analytics,pp\. 254–264\.Cited by:[2nd item](https://arxiv.org/html/2608.14211#A5.I1.i2.p1.1)\.
- \[3\]V\. Balazadeh, H\. Kamkari, V\. Thomas, B\. Li, J\. Ma, J\. C\. Cresswell, and R\. G\. Krishnan\(2025\)CausalPFN: Amortized Causal Effect Estimation via In\-Context Learning\.InAdvances in Neural Information Processing Systems,Cited by:[§3\.1](https://arxiv.org/html/2608.14211#S3.SS1.p1.1)\.
- \[4\]R\. Balestriero, M\. Ibrahim, V\. Sobal, A\. Morcos, S\. Shekhar, T\. Goldstein, F\. Bordes, A\. Bardes, G\. Mialon, Y\. Tian, A\. Schwarzschild, A\. G\. Wilson, J\. Geiping, Q\. Garrido, P\. Fernandez, A\. Bar, H\. Pirsiavash, Y\. LeCun, and M\. Goldblum\(2023\)A Cookbook of Self\-Supervised Learning\.arXiv:2304\.12210\.Cited by:[§3\.1](https://arxiv.org/html/2608.14211#S3.SS1.p1.1)\.
- \[5\]B\. Becker and R\. Kohavi\(1996\)Adult\.Note:UCI Machine Learning RepositoryDOI: https://doi\.org/10\.24432/C5XW20Cited by:[1st item](https://arxiv.org/html/2608.14211#A5.I1.i1.p1.1)\.
- \[6\]S\. Bird, M\. Dudík, R\. Edgar, B\. Horn, R\. Lutz, V\. Milan, M\. Sameki, H\. Wallach, and K\. Walker\(2020\)Fairlearn: A toolkit for assessing and improving fairness in AI\.Technical reportTechnical ReportMSR\-TR\-2020\-32,Microsoft\.Cited by:[§E\.4](https://arxiv.org/html/2608.14211#A5.SS4.p2.1),[§4\.2](https://arxiv.org/html/2608.14211#S4.SS2.p12.1)\.
- \[7\]T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei\(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.14211#S1.p1.1)\.
- \[8\]T\. Chen and C\. Guestrin\(2016\)XGBoost: A Scalable Tree Boosting System\.InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,External Links:ISBN 9781450342322,[Document](https://dx.doi.org/10.1145/2939672.2939785)Cited by:[§1](https://arxiv.org/html/2608.14211#S1.p1.1)\.
- \[9\]T\. Chen, S\. Kornblith, M\. Norouzi, and G\. Hinton\(2020\)A Simple Framework for Contrastive Learning of Visual Representations\.InProceedings of the 37th International Conference on Machine Learning,Vol\.119,pp\. 1597–1607\.Cited by:[§3\.1](https://arxiv.org/html/2608.14211#S3.SS1.p1.1)\.
- \[10\]J\. C\. Cresswell\(2025\)Trustworthy AI must account for interactions\.arXiv:2504\.07170\.Cited by:[Appendix A](https://arxiv.org/html/2608.14211#A1.p1.1)\.
- \[11\]A\. Defazio, X\. Yang, H\. Mehta, K\. Mishchenko, A\. Khaled, and A\. Cutkosky\(2024\)The road less scheduled\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 9974–10007\.External Links:[Document](https://dx.doi.org/10.52202/079017-0320)Cited by:[Appendix D](https://arxiv.org/html/2608.14211#A4.SS0.SSS0.Px2.p1.1)\.
- \[12\]F\. Ding, M\. Hardt, J\. Miller, and L\. Schmidt\(2021\)Retiring Adult: New Datasets for Fair Machine Learning\.InAdvances in Neural Information Processing Systems,Vol\.34\.Cited by:[§B\.1](https://arxiv.org/html/2608.14211#A2.SS1.p1.1),[§B\.2](https://arxiv.org/html/2608.14211#A2.SS2.p1.1),[§E\.6](https://arxiv.org/html/2608.14211#A5.SS6.p1.1),[§1](https://arxiv.org/html/2608.14211#S1.p5.1),[§4\.1](https://arxiv.org/html/2608.14211#S4.SS1.p1.1)\.
- \[13\]C\. Dwork, M\. Hardt, T\. Pitassi, O\. Reingold, and R\. Zemel\(2012\)Fairness through awareness\.InProceedings of the 3rd Innovations in Theoretical Computer Science Conference,External Links:ISBN 9781450311151,[Document](https://dx.doi.org/10.1145/2090236.2090255)Cited by:[1st item](https://arxiv.org/html/2608.14211#A3.I1.i1.p1.1)\.
- \[14\]N\. Erickson, L\. Purucker, A\. Tschalzev, D\. Holzmüller, P\. M\. Desai, D\. Salinas, and F\. Hutter\(2021\)TabArena: A Living Benchmark for Machine Learning on Tabular Data\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,Vol\.34\.Cited by:[§1](https://arxiv.org/html/2608.14211#S1.p1.1)\.
- \[15\]Y\. Ganin and V\. Lempitsky\(2015\)Unsupervised domain adaptation by backpropagation\.InProceedings of the 32nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.37,pp\. 1180–1189\.Cited by:[§2](https://arxiv.org/html/2608.14211#S2.p1.1),[§3\.2](https://arxiv.org/html/2608.14211#S3.SS2.p4.1)\.
- \[16\]Y\. Ganin, E\. Ustinova, H\. Ajakan, P\. Germain, H\. Larochelle, F\. Laviolette, M\. March, and V\. Lempitsky\(2016\)Domain\-Adversarial Training of Neural Networks\.Journal of Machine Learning Research17\(59\),pp\. 1–35\.Cited by:[§1](https://arxiv.org/html/2608.14211#S1.p4.1)\.
- \[17\]A\. Garg, M\. Ali, N\. Hollmann, L\. Purucker, S\. Müller, and F\. Hutter\(2025\)Real\-TabPFN: Improving Tabular Foundation Models via Continued Pre\-training With Real\-World Data\.arXiv:2507\.03971\.Cited by:[Appendix A](https://arxiv.org/html/2608.14211#A1.p3.1)\.
- \[18\]Y\. Gorishniy, I\. Rubachev, V\. Khrulkov, and A\. Babenko\(2021\)Revisiting deep learning models for tabular data\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 18932–18943\.Cited by:[Appendix A](https://arxiv.org/html/2608.14211#A1.p3.1)\.
- \[19\]L\. Grinsztajn, K\. Flöge, O\. Key, F\. Birkel, P\. Jund, B\. Roof, B\. Jäger, D\. Safaric, S\. Alessi, A\. Hayler, M\. Manium, R\. Yu, F\. Jablonski, S\. B\. Hoo, A\. Garg, J\. Robertson, M\. Bühler, V\. Moroshan, L\. Purucker, C\. Cornu, L\. C\. Wehrhahn, A\. Bonetto, B\. Schölkopf, S\. Gambhir, N\. Hollmann, and F\. Hutter\(2026\)TabPFN\-2\.5: Advancing the State of the Art in Tabular Foundation Models\.arXiv:2511\.08667\.Cited by:[Appendix A](https://arxiv.org/html/2608.14211#A1.p3.1),[§2](https://arxiv.org/html/2608.14211#S2.p2.1)\.
- \[20\]M\. Hardt, E\. Price, and N\. Srebro\(2016\)Equality of Opportunity in Supervised Learning\.InAdvances in Neural Information Processing Systems,Vol\.29\.Cited by:[Appendix A](https://arxiv.org/html/2608.14211#A1.p1.1),[2nd item](https://arxiv.org/html/2608.14211#A3.I1.i2.p1.1),[§2](https://arxiv.org/html/2608.14211#S2.p1.1)\.
- \[21\]N\. Hollmann, S\. Müller, K\. Eggensperger, and F\. Hutter\(2023\)TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second\.InInternational Conference on Learning Representations,Cited by:[Appendix A](https://arxiv.org/html/2608.14211#A1.p3.1),[§1](https://arxiv.org/html/2608.14211#S1.p1.1),[§2](https://arxiv.org/html/2608.14211#S2.p2.1)\.
- \[22\]N\. Hollmann, S\. Müller, L\. Purucker, A\. Krishnakumar, M\. Körfer, S\. B\. Hoo, R\. T\. Schirrmeister, and F\. Hutter\(2025\)Accurate predictions on small data with a tabular foundation model\.Nature637\(8045\),pp\. 319–326\.Cited by:[§1](https://arxiv.org/html/2608.14211#S1.p1.1),[§2](https://arxiv.org/html/2608.14211#S2.p2.1)\.
- \[23\]F\. Kamiran and T\. Calders\(2009\)Classifying without discriminating\.In2009 2nd International Conference on Computer, Control and Communication,pp\. 1–6\.External Links:[Document](https://dx.doi.org/10.1109/IC4.2009.4909197)Cited by:[Appendix A](https://arxiv.org/html/2608.14211#A1.p1.1),[§2](https://arxiv.org/html/2608.14211#S2.p1.1)\.
- \[24\]P\. Kenfack, S\. E\. Kahou, and U\. Aïvodji\(2026\)Towards fair in\-context learning with tabular foundation models\.Transactions on Machine Learning Research2026\.Cited by:[§1](https://arxiv.org/html/2608.14211#S1.p2.1)\.
- \[25\]M\. Khan, S\. Mohamed, B\. Marlin, and K\. Murphy\(2012\)A stick\-breaking likelihood for categorical data analysis with latent gaussian models\.InArtificial Intelligence and Statistics,pp\. 610–618\.Cited by:[§3\.1](https://arxiv.org/html/2608.14211#S3.SS1.p2.1)\.
- \[26\]M\. J\. Kusner, J\. Loftus, C\. Russell, and R\. Silva\(2017\)Counterfactual fairness\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[Appendix A](https://arxiv.org/html/2608.14211#A1.p4.1)\.
- \[27\]T\. Le Quy, A\. Roy, V\. Iosifidis, W\. Zhang, and E\. Ntoutsi\(2022\)A survey on datasets for fairness\-aware machine learning\.Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery12\(3\),pp\. e1452\.Cited by:[§E\.6](https://arxiv.org/html/2608.14211#A5.SS6.p1.1),[§2](https://arxiv.org/html/2608.14211#S2.p2.1)\.
- \[28\]I\. Loshchilov and F\. Hutter\(2019\)Decoupled Weight Decay Regularization\.InInternational Conference on Learning Representations,Cited by:[Appendix D](https://arxiv.org/html/2608.14211#A4.SS0.SSS0.Px2.p1.1)\.
- \[29\]J\. Ma, V\. Thomas, R\. Hosseinzadeh, A\. Labach, H\. Kamkari, J\. C\. Cresswell, K\. Golestan, G\. Yu, A\. L\. Caterini, and M\. Volkovs\(2025\)TabDPT: Scaling Tabular Foundation Models on Real Data\.InAdvances in Neural Information Processing Systems,Vol\.38\.Cited by:[Appendix A](https://arxiv.org/html/2608.14211#A1.p3.1),[§1](https://arxiv.org/html/2608.14211#S1.p1.1),[§2](https://arxiv.org/html/2608.14211#S2.p2.1),[§3\.1](https://arxiv.org/html/2608.14211#S3.SS1.p1.1)\.
- \[30\]D\. Madras, E\. Creager, T\. Pitassi, and R\. Zemel\(2018\)Learning adversarially fair and transferable representations\.InInternational Conference on Machine Learning,pp\. 3384–3393\.Cited by:[§E\.4](https://arxiv.org/html/2608.14211#A5.SS4.p2.1)\.
- \[31\]N\. Mehrabi, F\. Morstatter, N\. Saxena, K\. Lerman, and A\. Galstyan\(2021\)A Survey on Bias and Fairness in Machine Learning\.ACM Comput\. Surv\.54\(6\)\.External Links:ISSN 0360\-0300,[Document](https://dx.doi.org/10.1145/3457607)Cited by:[§1](https://arxiv.org/html/2608.14211#S1.p2.1)\.
- \[32\]F\. Pedregosa, G\. Varoquaux, A\. Gramfort, V\. Michel, B\. Thirion, O\. Grisel, M\. Blondel, P\. Prettenhofer, R\. Weiss, V\. Dubourg, J\. Vanderplas, A\. Passos, D\. Cournapeau, M\. Brucher, M\. Perrot, and É\. Duchesnay\(2011\)Scikit\-learn: Machine Learning in Python\.Journal of Machine Learning Research12,pp\. 2825–2830\.Cited by:[§4\.1](https://arxiv.org/html/2608.14211#S4.SS1.p2.1)\.
- \[33\]A\. Pfefferle, J\. Hog, L\. Purucker, and F\. Hutter\(2025\)nanoTabPFN: A Lightweight and Educational Reimplementation of TabPFN\.arXiv:2511\.03634\.Cited by:[§3\.2](https://arxiv.org/html/2608.14211#S3.SS2.p1.1),[§3\.2](https://arxiv.org/html/2608.14211#S3.SS2.p3.1)\.
- \[34\]J\. Qu, D\. Holzmüller, G\. Varoquaux, and M\. Le Morvan\(2025\)TabICL: A Tabular Foundation Model for In\-Context Learning on Large Data\.InProceedings of the 42nd International Conference on Machine Learning,Vol\.267,pp\. 50817–50847\.Cited by:[§2](https://arxiv.org/html/2608.14211#S2.p2.1),[§3\.3](https://arxiv.org/html/2608.14211#S3.SS3.p1.1)\.
- \[35\]J\. Qu, D\. Holzmüller, G\. Varoquaux, and M\. L\. Morvan\(2026\)TabICLv2: A better, faster, scalable, and open tabular foundation model\.InForty\-third International Conference on Machine Learning,Cited by:[Appendix A](https://arxiv.org/html/2608.14211#A1.p3.1),[Appendix D](https://arxiv.org/html/2608.14211#A4.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.14211#S1.p1.1),[§2](https://arxiv.org/html/2608.14211#S2.p2.1)\.
- \[36\]J\. Robertson, N\. Hollmann, S\. Müller, N\. Awad, and F\. Hutter\(2025\)FairPFN: A Tabular Foundation Model for Causal Fairness\.InForty\-second International Conference on Machine Learning,Cited by:[Appendix A](https://arxiv.org/html/2608.14211#A1.p4.1),[§2](https://arxiv.org/html/2608.14211#S2.p2.1),[§4\.1](https://arxiv.org/html/2608.14211#S4.SS1.p2.1)\.
- \[37\]C\. Stith, M\. Barath, V\. Balazadeh, J\. C\. Cresswell, and R\. G\. Krishnan\(2026\)Causal Foundation Models with Continuous Treatments\.arXiv:2605\.15133\.Cited by:[§3\.1](https://arxiv.org/html/2608.14211#S3.SS1.p1.1)\.
- \[38\]B\. Strack, J\. P\. DeShazo, C\. Gennings, J\. L\. Olmo, S\. Ventura, K\. J\. Cios, and J\. N\. Clore\(2014\)Impact of hba1c measurement on hospital readmission rates: analysis of 70,000 clinical database patient records\.BioMed research international2014\(1\),pp\. 781670\.Cited by:[4th item](https://arxiv.org/html/2608.14211#A5.I1.i4.p1.1)\.
- \[39\]Y\. Sui, T\. Wu, J\. Cresswell, G\. Wu, G\. Stein, X\. Huang, X\. Zhang, and M\. Volkovs\(2024\)Self\-supervised representation learning from random data projectors\.InInternational Conference on Learning Representations,Cited by:[§B\.2](https://arxiv.org/html/2608.14211#A2.SS2.p1.1)\.
- \[40\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§3\.2](https://arxiv.org/html/2608.14211#S3.SS2.p1.1)\.
- \[41\]L\. F\. Wightman\(1998\)LSAC national longitudinal bar passage study\. lsac research report series\.\.Cited by:[5th item](https://arxiv.org/html/2608.14211#A5.I1.i5.p1.1)\.
- \[42\]R\. Zemel, Y\. Wu, K\. Swersky, T\. Pitassi, and C\. Dwork\(2013\)Learning fair representations\.InProceedings of the 30th International Conference on Machine Learning,Vol\.28\.Cited by:[Appendix A](https://arxiv.org/html/2608.14211#A1.p1.1),[§E\.4](https://arxiv.org/html/2608.14211#A5.SS4.p2.1),[§2](https://arxiv.org/html/2608.14211#S2.p1.1),[§4\.2](https://arxiv.org/html/2608.14211#S4.SS2.p12.1)\.
- \[43\]B\. H\. Zhang, B\. Lemoine, and M\. Mitchell\(2018\)Mitigating unwanted biases with adversarial learning\.InProceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society,pp\. 335–340\.Cited by:[Appendix A](https://arxiv.org/html/2608.14211#A1.p1.1),[§2](https://arxiv.org/html/2608.14211#S2.p1.1)\.

## Appendix AAdditional Related Work

Fairness interventionsClassical fairness\-aware learning methods are often grouped by where the intervention occurs in the pipeline\[[10](https://arxiv.org/html/2608.14211#bib.bib8)\]\. Pre\-processing methods attempt to reduce bias directly in the data, for example through label massaging\[[23](https://arxiv.org/html/2608.14211#bib.bib25)\]or fair representation learning\[[42](https://arxiv.org/html/2608.14211#bib.bib24)\]\. In\-processing methods instead modify the training objective to encourage fairer behavior during optimization\[[43](https://arxiv.org/html/2608.14211#bib.bib27)\]\. Post\-processing methods operate on a trained model’s outputs to enforce fairness constraints after training\[[20](https://arxiv.org/html/2608.14211#bib.bib23)\]\. These families of methods have been influential, but they typically assume task\-specific model access, retraining, or output recalibration\.

Fairness in in\-context learningThese assumptions become restrictive for foundation models used through ICL, where predictions are generated without task\-specific parameter updates\. In our setting, the goal is not to retrofit fairness onto a frozen predictor after deployment, but to pretrain a model whose internal representations already support fairer predictions at inference time\. Our method therefore combines aspects of in\-processing and representation learning: fairness is encouraged during training, yet inference remains a single forward pass without additional optimization\.

Tabular foundation modelsWork on deep learning for tabular data has evolved from specialized supervised architectures\[[18](https://arxiv.org/html/2608.14211#bib.bib28)\]to pretrained tabular foundation models that generalize across tasks through ICL\. TabPFN\[[21](https://arxiv.org/html/2608.14211#bib.bib34)\]established this direction by showing that transformers trained on synthetic supervised tasks can perform strong zero\-shot prediction\. Subsequent models, including TabPFNv2\.5\[[19](https://arxiv.org/html/2608.14211#bib.bib4)\]and TabICLv2\[[35](https://arxiv.org/html/2608.14211#bib.bib1)\], improved efficiency and scale, while models such as TabDPT\[[29](https://arxiv.org/html/2608.14211#bib.bib11)\]and related real\-data pretraining approaches\[[17](https://arxiv.org/html/2608.14211#bib.bib29)\]sought to better align training with real\-world tabular distributions\.

Fairness\-aware TFMsAmong prior TFM work, FairPFN\[[36](https://arxiv.org/html/2608.14211#bib.bib17)\]is the closest to our setting because it also incorporates fairness during pretraining\. However, FairPFN is designed around a causal notion of fairness, using structurally generated datasets with biased and fair outcomes to target counterfactual fairness\[[26](https://arxiv.org/html/2608.14211#bib.bib30)\]\. Our focus is different: we study statistical group fairness notions, namely demographic parity, equal opportunity, and equalized odds, which are more directly aligned with the evaluation metrics commonly used in fair classification benchmarks\. Statistical fairness is data\-driven and focuses on distributional fairness \(outcome\-based\), while counterfactual fairness is more interventional and does not necessarily ensure outcome parity across groups

## Appendix BDatasets

In this section, we provide more details about the datasets used for evaluation\. We describe the prediction tasks and the dataset construction\.

### B\.1Prediction Task Details

We construct our benchmark from tasks provided by thefolktablesbenchmark\[[12](https://arxiv.org/html/2608.14211#bib.bib21)\], which is derived from the American Community Survey \(ACS\) Public Use Microdata Sample \(PUMS\)\. In particular, we consider five binary prediction tasks commonly used in prior work for fairness evaluation:

- •ACSIncome: predict whether an individual’s annual income exceeds $50,000\. Following the standard task definition, we restrict the data to individuals older than 16 who worked at least one hour per week during the previous year and earned at least $100\.
- •ACSMobility: predict whether an individual lived at the same address one year earlier\. We focus on individuals between 18 and 35 years old, which makes the task less imbalanced than in the full population, where most individuals do not move within a year\.
- •ACSTravelTime: predict whether an individual’s commute exceeds 20 minutes\. The task is defined on employed individuals older than 16, and the 20\-minute threshold roughly matches the median commute time in the 2018 ACS PUMS data\.
- •ACSEmployment: predict whether an individual is employed\. For this task, we consider individuals between 16 and 90 years old\.
- •ACSPublicCoverage: predict whether an individual receives public health insurance coverage\. We restrict the sample to individuals younger than 65 with income below $30,000, thereby focusing on lower\-income individuals who are not eligible for Medicare\.

### B\.2Task Construction

Generating new learning signals from randomly selected features in pretraining data is a core concept in self\-supervised learning\[[39](https://arxiv.org/html/2608.14211#bib.bib7)\]\. In our work, we specifically construct fairness tasks, which is necessary to scale up pretraining, due to insufficient existing datasets labeled with fairness attributes\. For each base prediction task, we instantiate fairness evaluation settings by combining it with three sensitive attributes—Gender, Age, and Race—and with data drawn from eight states: Alabama \(AL\), California \(CA\), Hawaii \(HI\), Indiana \(IN\), Maine \(ME\), Michigan \(MI\), New Mexico \(NM\), and New York \(NY\)\. We choose these states to span a range of bias levels reported in the originalfolktablesstudy\[[12](https://arxiv.org/html/2608.14211#bib.bib21)\], so that the benchmark includes settings with meaningfully different fairness profiles rather than a narrow slice of the ACS\. This yields a total of5×3×8=1205\\times 3\\times 8=120tasks\. In other words, each prediction problem contributes 24 tasks, each sensitive attribute appears in 40 tasks, and each state contributes 15 tasks to the full benchmark\. For the sensitive attributes, Age is binarized using a 25\-year\-old threshold, and Race is restricted to White and Black Americans\. Across all task instantiations, dataset sizes vary from roughly 3\.5k to 12k samples, and we use a random 80/20 train–test split\.

This evaluation design is important for two reasons\. First, varying the prediction problem, sensitive attribute, and state allows us to assess fairness across substantially different label distributions, demographic compositions, and regional contexts, rather than tailoring conclusions to a single task configuration\. Second, reporting results over the full Cartesian product reduces the risk that observed fairness improvements are driven by a small number of favorable settings\. We therefore view performance aggregated over these 120 tasks as a stronger indicator of whether a method learns fairness\-aware behavior that transfers across heterogeneous real\-world tabular prediction problems\.

## Appendix CFairness metrics

In this work, we focus on group fairness criteria that quantify disparities in model behavior across demographic groups\. LetY^=f⁡\(X\)\\hat\{Y\}=f\(X\)denote the binary prediction of a classifier, letY∈\{0,1\}Y\\in\\\{0,1\\\}be the ground\-truth label, and letS∈\{0,1\}S\\in\\\{0,1\\\}denote the sensitive attribute\. We consider the following three standard fairness notions\.

- •Demographic parity \(DP\)requires the rate of positive predictions to be the same across groups\[[13](https://arxiv.org/html/2608.14211#bib.bib22)\]\. Formally, ℙ⁡\(Y^=1∣S=0\)=ℙ⁡\(Y^=1∣S=1\)\.\\mathbb\{P\}\(\\hat\{Y\}=1\\mid S=0\)=\\mathbb\{P\}\(\\hat\{Y\}=1\\mid S=1\)\.\(2\)
- •Equalized odds \(EOD\)requires the predictor to have the same true positive and false positive rates across groups\[[20](https://arxiv.org/html/2608.14211#bib.bib23)\]\. Equivalently, for each label valuey∈\{0,1\}y\\in\\\{0,1\\\}, ℙ⁡\(Y^=1∣S=0,Y=y\)=ℙ⁡\(Y^=1∣S=1,Y=y\)\.\\mathbb\{P\}\(\\hat\{Y\}=1\\mid S=0,Y=y\)=\\mathbb\{P\}\(\\hat\{Y\}=1\\mid S=1,Y=y\)\.\(3\)
- •Equal opportunity \(EOP\)focuses only on parity of true positive rates across groups\. It can be viewed as they=1y=1special case of equalized odds: ℙ⁡\(Y^=1∣S=0,Y=1\)=ℙ⁡\(Y^=1∣S=1,Y=1\)\.\\mathbb\{P\}\(\\hat\{Y\}=1\\mid S=0,Y=1\)=\\mathbb\{P\}\(\\hat\{Y\}=1\\mid S=1,Y=1\)\.\(4\)

In the experiments, we report empirical disparity versions of these metrics\. For demographic parity, we use the absolute difference in the expected positive prediction rate across groups:

DP=\|𝔼x\|S=0\[𝕀\{Y^=1\}\]−𝔼x\|S=1\[𝕀\{Y^=1\}\]\|\.\\mathrm\{DP\}=\\left\|\\underset\{x\\mid S=0\}\{\\mathbb\{E\}\}\\left\[\\mathbb\{I\}\\\{\\hat\{Y\}=1\\\}\\right\]\-\\underset\{x\\mid S=1\}\{\\mathbb\{E\}\}\\left\[\\mathbb\{I\}\\\{\\hat\{Y\}=1\\\}\\right\]\\right\|\.\(5\)Where𝕀⁡\(⋅\)\\mathbb\{I\}\(\\cdot\)denotes the indicator function\.

For the equalized\-odds\-based metrics, we define the group gaps in false positive rate and true positive rate using the same expectation notation:

δFPR=\|𝔼x∣S=0,Y=0\[𝕀\{Y^=1\}\]−𝔼x∣S=1,Y=0\[𝕀\{Y^=1\}\]\|,\\delta\_\{\\mathrm\{FPR\}\}=\\left\|\\underset\{x\\mid S=0,Y=0\}\{\\mathbb\{E\}\}\\left\[\\mathbb\{I\}\\\{\\hat\{Y\}=1\\\}\\right\]\-\\underset\{x\\mid S=1,Y=0\}\{\\mathbb\{E\}\}\\left\[\\mathbb\{I\}\\\{\\hat\{Y\}=1\\\}\\right\]\\right\|,\(6\)δTPR=\|𝔼x∣S=0,Y=1\[𝕀\{Y^=1\}\]−𝔼x∣S=1,Y=1\[𝕀\{Y^=1\}\]\|\.\\delta\_\{\\mathrm\{TPR\}\}=\\left\|\\underset\{x\\mid S=0,Y=1\}\{\\mathbb\{E\}\}\\left\[\\mathbb\{I\}\\\{\\hat\{Y\}=1\\\}\\right\]\-\\underset\{x\\mid S=1,Y=1\}\{\\mathbb\{E\}\}\\left\[\\mathbb\{I\}\\\{\\hat\{Y\}=1\\\}\\right\]\\right\|\.\(7\)We then report

EOD=max⁡\(δFPR,δTPR\)\\mathrm\{EOD\}=\\max\\left\(\\delta\_\{\\mathrm\{FPR\}\},\\delta\_\{\\mathrm\{TPR\}\}\\right\)\(8\)EOP=δTPR\.\\mathrm\{EOP\}=\\delta\_\{\\mathrm\{TPR\}\}\.\(9\)
Smaller values ofDP\\mathrm\{DP\},EOD\\mathrm\{EOD\}, andEOP\\mathrm\{EOP\}indicate fairer behaviour, with zero corresponding to perfect parity under the respective criterion\. We use these empirical gaps because they provide an interpretable summary of group\-level disparities and are standard in fairness evaluations for binary classification\.

## Appendix DModel Architecture and Hyperparameters

Our model builds on the nanoTabPFN architecture fromTFM\-Playground222https://github\.com/automl/TFM\-Playground, which we use as a lightweight transformer backbone for fairness\-aware pretraining\. Concretely, we use a model with 6 transformer layers, 6 attention heads, embedding dimension 192, and feed\-forward hidden dimension 192\. Consistent with the main architecture described in Section 3, we augment this backbone with three input encoders—for features, targets, and sensitive attributes—that map their respective inputs into the shared 192\-dimensional token space\. The label\-prediction head and the sensitive\-attribute head are both implemented as two\-layer MLPs with hidden size 768\.

#### Pretraining setup\.

We pretrain on 300,000 synthetically generated tabular datasets sampled from the TabICL’s prior implementation\[[35](https://arxiv.org/html/2608.14211#bib.bib1)\]\. Each sampled task contains 150 datapoints, 6 features, and 2 classes, and training is performed with batch size 32\. During pretraining, the model receives context and query sets as described in Section 3, with the query label masked and the query sensitive attribute replaced by the learned mask token\. All reportedFairTFMresults are obtained from checkpoints trained under this shared setup, varying only the fairness weightλ\\lambdain the joint objective\.

#### Optimization details\.

We optimize the model using Schedule\-Free AdamW\[[11](https://arxiv.org/html/2608.14211#bib.bib37),[28](https://arxiv.org/html/2608.14211#bib.bib38)\]with learning rate1×10−41\\times 10^\{\-4\}and no weight decay\. This choice provided stable optimization across the fairness weights considered in the main experiments\. For the fairness\-aware variants, the only task\-level hyperparameter we vary isλ\\lambda, which directly controls the strength of the adversarial sensitive\-attribute objective and thereby the fairness–accuracy trade\-off\.

## Appendix EAdditional Results

### E\.1Aggregated results on the 120 ACS tasks

Table[2](https://arxiv.org/html/2608.14211#A5.T2)reports the aggregated accuracy and fairness metrics behind Figure[2](https://arxiv.org/html/2608.14211#S4.F2), and Figure[7](https://arxiv.org/html/2608.14211#A5.F7)shows the corresponding Pareto front when predictive performance is measured with AUCROC instead of accuracy\. The two views agree: unconstrained TFMs sit in the most accurate but least fair region, whileFairTFMvariants trace the fair end of the front\.

Figure 7:Pareto front between AUCROC and fairness for various models\.↑\\uparrowindicates higher is better \(accuracy\) and↓\\downarrowindicates smaller is better \(unfairness\)\.Table 2:Average accuracy, AUCROC, and fairness metrics across the 120 benchmark tasks, reported as mean±\\pmstandard deviation over three random seeds\. This table complements Figure[2](https://arxiv.org/html/2608.14211#S4.F2)by summarizing the final\-checkpoint performance ofFairTFMmodels\. Recent unconstrained TFMs attain the highest average accuracy but also the largest fairness gaps, classical baselines are generally less accurate without being substantially fairer, andFairTFMvariants provide better fairness–accuracy trade\-offs, with largerλ\\lambdayielding progressively lower unfairness at the cost of reduced accuracy\.ModelAccuracyAUCROCDP DiffEOD DiffEOP DiffTabICLv20\.771±0\.05\\mathbf\{0\.771\\pm 0\.05\}0\.802±0\.07\\mathbf\{0\.802\\pm 0\.07\}0\.120±0\.110\.120\\pm 0\.110\.140±0\.100\.140\\pm 0\.100\.096±0\.090\.096\\pm 0\.09TabPFNv2\.50\.770±0\.050\.770\\pm 0\.050\.801±0\.070\.801\\pm 0\.070\.121±0\.120\.121\\pm 0\.120\.145±0\.110\.145\\pm 0\.110\.098±0\.100\.098\\pm 0\.10XGB0\.744±0\.060\.744\\pm 0\.060\.760±0\.090\.760\\pm 0\.090\.116±0\.110\.116\\pm 0\.110\.144±0\.110\.144\\pm 0\.110\.097±0\.100\.097\\pm 0\.10KNN0\.716±0\.070\.716\\pm 0\.070\.707±0\.100\.707\\pm 0\.100\.106±0\.110\.106\\pm 0\.110\.139±0\.100\.139\\pm 0\.100\.093±0\.090\.093\\pm 0\.09RF0\.745±0\.060\.745\\pm 0\.060\.767±0\.080\.767\\pm 0\.080\.113±0\.110\.113\\pm 0\.110\.147±0\.110\.147\\pm 0\.110\.100±0\.100\.100\\pm 0\.10LR0\.728±0\.060\.728\\pm 0\.060\.733±0\.100\.733\\pm 0\.100\.107±0\.120\.107\\pm 0\.120\.134±0\.110\.134\\pm 0\.110\.094±0\.090\.094\\pm 0\.09FairPFN0\.690±0\.070\.690\\pm 0\.070\.695±0\.110\.695\\pm 0\.110\.043±0\.07\\mathbf\{0\.043\\pm 0\.07\}0\.078±0\.110\.078\\pm 0\.110\.069±0\.100\.069\\pm 0\.10FairTFM\-0\.70\.729±0\.070\.729\\pm 0\.070\.754±0\.100\.754\\pm 0\.100\.077±0\.130\.077\\pm 0\.130\.098±0\.130\.098\\pm 0\.130\.065±0\.100\.065\\pm 0\.10FairTFM\-1\.00\.726±0\.070\.726\\pm 0\.070\.751±0\.100\.751\\pm 0\.100\.075±0\.120\.075\\pm 0\.120\.091±0\.120\.091\\pm 0\.120\.064±0\.090\.064\\pm 0\.09FairTFM\-100\.711±0\.070\.711\\pm 0\.070\.743±0\.100\.743\\pm 0\.100\.063±0\.120\.063\\pm 0\.120\.076±0\.120\.076\\pm 0\.120\.051±0\.080\.051\\pm 0\.08FairTFM\-250\.688±0\.060\.688\\pm 0\.060\.731±0\.100\.731\\pm 0\.100\.059±0\.140\.059\\pm 0\.140\.072±0\.15\\mathbf\{0\.072\\pm 0\.15\}0\.025±0\.06\\mathbf\{0\.025\\pm 0\.06\}
### E\.2Percent improvement over the strongest baselines

To make the size of the fairness gains explicit, Table[3](https://arxiv.org/html/2608.14211#A5.T3)reports the relative change of eachFairTFMvariant against the strongest TFM baseline \(TabPFNv2\.5\), the strongest classical baseline \(XGB\), and LR, computed from the averages in Table[2](https://arxiv.org/html/2608.14211#A5.T2)\. Positive fairness numbers mean fairer \(lower DP, EOD, or EOP\), and negative accuracy and AUCROC numbers mean lower predictive performance\. Three points stand out\. First, the fairness gains are large relative to the predictive costs:FairTFM\-0\.7 improves DP, EOD, and EOP by 32–36% for a 5\.3% accuracy cost against TabPFNv2\.5, and only a 2% accuracy and 0\.8% AUCROC cost against XGB\. Second, against LR—the classical baseline closest in accuracy—FairTFM\-0\.7 improves every metric at once: slightly higher accuracy \(\+0\.1%\), higher AUCROC \(\+2\.9%\), and 27–31% better fairness\. Third, the gains grow monotonically withλ\\lambda, up to 51–75% forFairTFM\-25, confirming thatλ\\lambdaprovides direct control over the fairness–accuracy trade\-off\.

Table 3:Fairness improvement \(positive numbers mean fairer\) and accuracy and AUCROC change of eachFairTFMvariant relative to \(a\) the best TFM \(TabPFNv2\.5\), \(b\) the best classical model \(XGB\), and \(c\) LR, averaged over the 120 benchmark tasks and derived from Table[2](https://arxiv.org/html/2608.14211#A5.T2)\.\(a\)vs\. TabPFNv2\.5ModelAccAUCROCDPEODEOPFairTFM\-0\.7−5\.3%\-5\.3\\%−5\.9%\-5\.9\\%\+36\.4%\\mathbf\{\+36\.4\\%\}\+32\.4%\\mathbf\{\+32\.4\\%\}\+33\.7%\\mathbf\{\+33\.7\\%\}FairTFM\-1\.0−5\.7%\-5\.7\\%−6\.2%\-6\.2\\%\+38\.0%\\mathbf\{\+38\.0\\%\}\+37\.2%\\mathbf\{\+37\.2\\%\}\+34\.7%\\mathbf\{\+34\.7\\%\}FairTFM\-10−7\.7%\-7\.7\\%−7\.2%\-7\.2\\%\+47\.9%\\mathbf\{\+47\.9\\%\}\+47\.6%\\mathbf\{\+47\.6\\%\}\+48\.0%\\mathbf\{\+48\.0\\%\}FairTFM\-25−10\.6%\-10\.6\\%−8\.7%\-8\.7\\%\+51\.2%\\mathbf\{\+51\.2\\%\}\+50\.3%\\mathbf\{\+50\.3\\%\}\+74\.5%\\mathbf\{\+74\.5\\%\}
\(b\)vs\. XGBModelAccAUCROCDPEODEOPFairTFM\-0\.7−2\.0%\-2\.0\\%−0\.8%\-0\.8\\%\+33\.6%\\mathbf\{\+33\.6\\%\}\+31\.9%\\mathbf\{\+31\.9\\%\}\+33\.0%\\mathbf\{\+33\.0\\%\}FairTFM\-1\.0−2\.4%\-2\.4\\%−1\.2%\-1\.2\\%\+35\.3%\\mathbf\{\+35\.3\\%\}\+36\.8%\\mathbf\{\+36\.8\\%\}\+34\.0%\\mathbf\{\+34\.0\\%\}FairTFM\-10−4\.4%\-4\.4\\%−2\.2%\-2\.2\\%\+45\.7%\\mathbf\{\+45\.7\\%\}\+47\.2%\\mathbf\{\+47\.2\\%\}\+47\.4%\\mathbf\{\+47\.4\\%\}FairTFM\-25−7\.5%\-7\.5\\%−3\.8%\-3\.8\\%\+49\.1%\\mathbf\{\+49\.1\\%\}\+50\.0%\\mathbf\{\+50\.0\\%\}\+74\.2%\\mathbf\{\+74\.2\\%\}
\(c\)vs\. LRModelAccAUCROCDPEODEOPFairTFM\-0\.7\+0\.1%\+0\.1\\%\+2\.9%\+2\.9\\%\+28\.0%\\mathbf\{\+28\.0\\%\}\+26\.9%\\mathbf\{\+26\.9\\%\}\+30\.9%\\mathbf\{\+30\.9\\%\}FairTFM\-1\.0−0\.3%\-0\.3\\%\+2\.5%\+2\.5\\%\+29\.9%\\mathbf\{\+29\.9\\%\}\+32\.1%\\mathbf\{\+32\.1\\%\}\+31\.9%\\mathbf\{\+31\.9\\%\}FairTFM\-10−2\.3%\-2\.3\\%\+1\.4%\+1\.4\\%\+41\.1%\\mathbf\{\+41\.1\\%\}\+43\.3%\\mathbf\{\+43\.3\\%\}\+45\.7%\\mathbf\{\+45\.7\\%\}FairTFM\-25−5\.5%\-5\.5\\%−0\.3%\-0\.3\\%\+44\.9%\\mathbf\{\+44\.9\\%\}\+46\.3%\\mathbf\{\+46\.3\\%\}\+73\.4%\\mathbf\{\+73\.4\\%\}

### E\.3Performance dynamics during training

Figure[4](https://arxiv.org/html/2608.14211#S4.F4)in the main text provides a finer\-grained view of how fairness emerges over the course of pretraining\. We track accuracy and fairness \(DP and EOD\) forFairTFMwithλ∈\{0\.7,25\}\\lambda\\in\\\{0\.7,25\\\}and for nanoTabPFN, evaluating checkpoints throughout training and averaging the resulting curves over 10 real\-world fairness tasks from our benchmark\. The shaded regions indicate one standard deviation across these tasks\. Since lower DP and EOD correspond to fairer behavior, the middle and right panels show that bothFairTFMvariants achieve a fairness advantage over nanoTabPFN very early in training and retain this advantage throughout optimization\. This gap is not confined to a narrow set of checkpoints: it persists across most of the training trajectory and becomes especially pronounced in the later stages\. We note that the nanoTabPFN reference is trained under the same data prior, backbone, optimizer, and training budget asFairTFM; the two differ only in the fairness\-specific components \(the sensitive\-feature designation in task sampling, the added encoder and adversarial head, and the fairness loss\), so this comparison isolates the effect of our intervention\.

At the same time, the figure makes the fairness–accuracy trade\-off induced byλ\\lambdavisually explicit\. Specifically, nanoTabPFN andFairTFMwithλ=0\.7\\lambda=0\.7converge to very similar final accuracies, but the latter does so while maintaining consistently lower DP and EOD gaps\. By contrast,FairTFMwithλ=25\\lambda=25occupies a distinctly different operating regime: its accuracy remains below that of the other two models, but it attains by far the lowest unfairness on both metrics\. The trajectories are also mildly non\-monotonic, which is expected because fairness and accuracy are measured on heterogeneous downstream tasks rather than on the pretraining objective itself\. Taken together, these dynamics show that our objective changes the optimization path, steering the encoder toward representations that are progressively less informative about the sensitive attribute, withλ\\lambdaproviding a direct handle on the final fairness–accuracy operating point\.

### E\.4Fair representation learning: protocol details and additional results

The fairness condition in Definition[1](https://arxiv.org/html/2608.14211#Thmdefinition1)is equivalent toℙ⁡\(Z∣Y,S\)=ℙ⁡\(Z∣Y\)\\mathbb\{P\}\(Z\\mid Y,S\)=\\mathbb\{P\}\(Z\\mid Y\)or, in information\-theoretic terms,I⁡\(Z;S∣Y\)=0I\(Z;S\\mid Y\)=0\. A downstream model trained onZZis therefore expected to exhibit improved fairness relative to a model trained directly onXX\.

Existing approaches for learningggare typically task\-specific, requiring a separate model to be fitted for each downstream task\[[42](https://arxiv.org/html/2608.14211#bib.bib24),[30](https://arxiv.org/html/2608.14211#bib.bib36)\]\. We compareFairTFMagainst two open\-source, sklearn\-compatible pre\-processing methods from thefairlearnlibrary\[[6](https://arxiv.org/html/2608.14211#bib.bib35)\]\. The first is Correlation Remover \(CR\), which reduces linear dependence on the sensitive attribute by applying a linear transformation to the non\-sensitive features\[[6](https://arxiv.org/html/2608.14211#bib.bib35)\]\. The second is Learning Fair Representations \(LFR\), which maps inputs to latent prototypes while encouraging similar assignment behaviour across demographic groups\[[42](https://arxiv.org/html/2608.14211#bib.bib24)\]\.

We consider ACSIncome for Alabama \(AL\) with three sensitive attributes—gender, race, and age—yielding three downstream fairness tasks\. Each task is randomly split into 80% training data and 20% test data\. For CR and LFR, the training split is used to fit the representation mapgg, which is then applied to both the training and test features\. ForFairTFM, we use the training split as context with the test as query, and compute embeddings for both training and test examples from the transformer’s output\. Importantly, these representations are obtained in a single forward pass, without any task\-specific gradient updates\.

We train three downstream classifiers—LR, RF, and XGB—on four types of representations: the raw input features and the fair representations produced by CR, LFR, andFairTFM\. For CR and LFR, we sweep the parameter controlling the fairness–accuracy trade\-off over\[0\.01,0\.02,…,0\.1,0\.2,…,1\.0\]\[0\.01,0\.02,\\ldots,0\.1,0\.2,\\ldots,1\.0\], with denser coverage in the higher\-fairness regime\. ForFairTFM, we evaluate the checkpoints associated with theλ\\lambdavalues used in Figure[2](https://arxiv.org/html/2608.14211#S4.F2)\. We run this experiment across three random seeds and average the results\.

Figure[8](https://arxiv.org/html/2608.14211#A5.F8)shows the XGBoost results for the remaining sensitive attributes \(race and age; gender is in Figure[6](https://arxiv.org/html/2608.14211#S4.F6)of the main text\), and Figures[9](https://arxiv.org/html/2608.14211#A5.F9)and[10](https://arxiv.org/html/2608.14211#A5.F10)show the Pareto fronts with Random Forest and Logistic Regression as downstream models\. Across all three downstream models,FairTFMembeddings provide competitive fairness–accuracy trade\-offs without any task\-specific fitting of the representation map, while CR is most effective for the linear LR model, as it only removes linear correlation with the sensitive attribute\.

\(a\)Race\(b\)Age
Figure 8:Pareto front of XGBoost models trained with different data representations, for race and age as sensitive attributes \(gender is shown in Figure[6](https://arxiv.org/html/2608.14211#S4.F6)of the main text\)\.\(a\)Gender\(b\)Race\(c\)Age
Figure 9:Pareto front of Random Forest models trained with different data representations\.\(a\)Gender\(b\)Race\(c\)Age
Figure 10:Pareto front of Logistic Regression models trained with different data representations\.
### E\.5Additional Exponentiated Gradient results

Figure[11](https://arxiv.org/html/2608.14211#A5.F11)complements Figure[5](https://arxiv.org/html/2608.14211#S4.F5)in the main text by showing the fairness–accuracy trade\-off against the EG\-constrained baselines when race and age are used as sensitive attributes \(same setup, ACSIncome in Alabama\)\. For race, where all methods incur larger fairness gaps,FairTFMspans a wide and competitive portion of the frontier, particularly at the lower\-unfairness end\. For age, EG\-based classical models achieve stronger accuracy–fairness trade\-offs for EOD and EOP, whereasFairTFMremains competitive on DP\.

\(a\)Race\(b\)Age
Figure 11:Fairness\-accuracy trade\-off comparison with task\-specific classical models trained with fairness constraints using Exponentiated Gradient \(EG\), on ACSIncome \(Alabama\) with race and age as sensitive attributes \(gender is shown in Figure[5](https://arxiv.org/html/2608.14211#S4.F5)of the main text\)\.Figure[12](https://arxiv.org/html/2608.14211#A5.F12)additionally measures predictive performance with AUCROC instead of accuracy\. Because the EG reduction produces randomized ensembles that trade calibrated scores for constraint satisfaction, its AUCROC drops sharply, andFairTFMprovides the best overall Pareto front across all three sensitive attributes under this view\.

\(a\)Gender\(b\)Race\(c\)Age
Figure 12:Pareto front of classical models trained with fairness constraints using Exponentiated Gradient \(EG\)\.FairTFMshow best overall pareto front when performance are measured with AUCROC\.
### E\.6Evaluation beyond ACS PUMS tasks

The main evaluation covers 120 tasks derived from ACS PUMS\[[12](https://arxiv.org/html/2608.14211#bib.bib21)\]\. To test whether the observed fairness–utility trends extend beyond this data source, we additionally evaluate on a collection of widely used fairness benchmarks spanning different domains, prediction targets, and sensitive attributes\[[27](https://arxiv.org/html/2608.14211#bib.bib19)\]\. Specifically, we consider the following datasets:

- •The Adult \(Census Income\) dataset contains demographic and socioeconomic records from the U\.S\. Census\[[5](https://arxiv.org/html/2608.14211#bib.bib42)\]\. The task is to predict whether an individual’s annual income exceeds $50,000, using gender, race, and age as sensitive attributes\.
- •The COMPAS dataset contains criminal justice screening records for defendants\[[2](https://arxiv.org/html/2608.14211#bib.bib40)\]\. The task is to predict whether an individual will be rearrested within two years of their initial arrest, with race as the sensitive attribute\.
- •The German Credit dataset contains records of bank account holders and is commonly used for credit risk assessment\. The task is to classify applicants as low\-risk or high\-risk, using gender and age \(≤\\leq25 years\) as sensitive attributes\.
- •The Diabetes dataset contains clinical records from 130 U\.S\. hospitals collected between 1999 and 2008\[[38](https://arxiv.org/html/2608.14211#bib.bib41)\]\. The task is to predict whether a patient will be readmitted within 30 days of discharge, using gender and race as sensitive attributes\.
- •The Law School dataset consists of admissions records collected by the Law School Admission Council \(LSAC\) from 163 U\.S\. law schools in 1991\[[41](https://arxiv.org/html/2608.14211#bib.bib39)\]\. The task is to predict whether a candidate will pass the bar exam, using race and gender as sensitive attributes\.
- •The CelebA dataset provides facial\-attribute annotations for celebrity images\. We consider two binary prediction tasks,Blond HairandSmiling, using gender as the sensitive attribute in both cases\.

Together, these benchmarks define 12 additional fairness tasks\. We use the same evaluation protocol as in Section[4](https://arxiv.org/html/2608.14211#S4)and average results over three random seeds\. Figure[3](https://arxiv.org/html/2608.14211#S4.F3)in the main text shows the accuracy Pareto fronts, Figure[13](https://arxiv.org/html/2608.14211#A5.F13)below shows the AUCROC view, and Table[4](https://arxiv.org/html/2608.14211#A5.T4)reports the averaged metrics, playing the same role as Table[2](https://arxiv.org/html/2608.14211#A5.T2)for the 120 ACS tasks\. As there, the table reports the last pretraining checkpoint of eachFairTFMvariant, so it is a conservative summary: Pareto\-dominant checkpoints in the figures could yield better trade\-offs, at the cost of model selection on a real\-world validation set\. The trend from the main evaluation transfers fully to these tasks:FairTFM\-0\.7 reduces DP by 49% and EOD by 47% relative to TabPFNv2\.5 while matching XGB and LR in AUCROC \(0\.803\), andFairTFM\-25 nearly eliminates group disparities \(DP of 0\.011, an 89% reduction\)\. Notably, KNN—the fairest unconstrained baseline—achieves its lower disparities only through much weaker predictive performance \(AUCROC 0\.664\), whereasFairTFMoffers strictly better fairness at substantially higher AUCROC\. SinceFairTFMis pretrained on purely synthetic data, all real datasets are unseen during pretraining; these results additionally rule out any ACS\-specific effect in the evaluation and show that the fairness behavior learned from synthetic pretraining generalizes across data provenances\.

Table 4:Average fairness and accuracy metrics across the 12 fairness tasks beyond ACS PUMS, reported as mean±\\pmstandard deviation over three random seeds\. The same pattern as on the 120 ACS tasks holds: unconstrained TFMs are the most accurate but least fair, whileFairTFMvariants sharply reduce disparities at a moderate predictive cost\.ModelAccuracyAUCROCDP DiffEOD DiffEOP DiffTabICLv20\.796±0\.09\\mathbf\{0\.796\\pm 0\.09\}0\.829±0\.09\\mathbf\{0\.829\\pm 0\.09\}0\.101±0\.090\.101\\pm 0\.090\.097±0\.060\.097\\pm 0\.060\.087±0\.060\.087\\pm 0\.06TabPFNv2\.50\.793±0\.100\.793\\pm 0\.100\.828±0\.090\.828\\pm 0\.090\.097±0\.090\.097\\pm 0\.090\.106±0\.070\.106\\pm 0\.070\.092±0\.080\.092\\pm 0\.08KNN0\.732±0\.100\.732\\pm 0\.100\.664±0\.140\.664\\pm 0\.140\.065±0\.060\.065\\pm 0\.060\.078±0\.050\.078\\pm 0\.050\.062±0\.060\.062\\pm 0\.06XGB0\.780±0\.100\.780\\pm 0\.100\.803±0\.100\.803\\pm 0\.100\.100±0\.090\.100\\pm 0\.090\.099±0\.070\.099\\pm 0\.070\.085±0\.070\.085\\pm 0\.07LR0\.784±0\.090\.784\\pm 0\.090\.803±0\.100\.803\\pm 0\.100\.099±0\.090\.099\\pm 0\.090\.115±0\.080\.115\\pm 0\.080\.087±0\.070\.087\\pm 0\.07RF0\.786±0\.090\.786\\pm 0\.090\.806±0\.090\.806\\pm 0\.090\.102±0\.090\.102\\pm 0\.090\.112±0\.060\.112\\pm 0\.060\.084±0\.050\.084\\pm 0\.05FairPFN0\.737±0\.100\.737\\pm 0\.100\.705±0\.100\.705\\pm 0\.100\.045±0\.080\.045\\pm 0\.080\.062±0\.090\.062\\pm 0\.090\.043±0\.080\.043\\pm 0\.08FairTFM\-0\.10\.753±0\.110\.753\\pm 0\.110\.782±0\.130\.782\\pm 0\.130\.052±0\.090\.052\\pm 0\.090\.060±0\.100\.060\\pm 0\.100\.060±0\.100\.060\\pm 0\.10FairTFM\-0\.70\.752±0\.100\.752\\pm 0\.100\.803±0\.090\.803\\pm 0\.090\.049±0\.080\.049\\pm 0\.080\.056±0\.090\.056\\pm 0\.090\.056±0\.090\.056\\pm 0\.09FairTFM\-100\.734±0\.100\.734\\pm 0\.100\.780±0\.100\.780\\pm 0\.100\.021±0\.070\.021\\pm 0\.070\.025±0\.070\.025\\pm 0\.070\.025±0\.070\.025\\pm 0\.07FairTFM\-250\.708±0\.140\.708\\pm 0\.140\.737±0\.120\.737\\pm 0\.120\.011±0\.10\\mathbf\{0\.011\\pm 0\.10\}0\.015±0\.10\\mathbf\{0\.015\\pm 0\.10\}0\.015±0\.10\\mathbf\{0\.015\\pm 0\.10\}Figure 13:Pareto front between AUCROC and fairness for various models on the 12 fairness tasks beyond ACS PUMS\.↑\\uparrowindicates higher is better and↓\\downarrowindicates smaller is better \(unfairness\)\.As for the 120 ACS tasks, Table[5](https://arxiv.org/html/2608.14211#A5.T5)reports the relative change of eachFairTFMvariant against TabPFNv2\.5, XGB, and LR, derived from Table[4](https://arxiv.org/html/2608.14211#A5.T4)\. The picture is even stronger than on the ACS benchmark:FairTFM\-0\.7 improves DP, EOD, and EOP by 39–50% against TabPFNv2\.5 for a 5\.2% accuracy and 3\.0% AUCROC cost, and against XGB and LR it improves fairness by 34–51% with*no*AUCROC cost at all \(0\.803 for all three models\)\. At the fair end,FairTFM\-25 improves fairness by 82–89% across all three baselines\.

Table 5:Fairness improvement \(positive numbers mean fairer\) and accuracy and AUCROC change of eachFairTFMvariant relative to \(a\) TabPFNv2\.5, \(b\) XGB, and \(c\) LR, averaged over the 12 fairness tasks beyond ACS PUMS and derived from Table[4](https://arxiv.org/html/2608.14211#A5.T4)\.\(a\)vs\. TabPFNv2\.5ModelAccAUCROCDPEODEOPFairTFM\-0\.1−5\.0%\-5\.0\\%−5\.6%\-5\.6\\%\+46\.4%\\mathbf\{\+46\.4\\%\}\+43\.4%\\mathbf\{\+43\.4\\%\}\+34\.8%\\mathbf\{\+34\.8\\%\}FairTFM\-0\.7−5\.2%\-5\.2\\%−3\.0%\-3\.0\\%\+49\.5%\\mathbf\{\+49\.5\\%\}\+47\.2%\\mathbf\{\+47\.2\\%\}\+39\.1%\\mathbf\{\+39\.1\\%\}FairTFM\-10−7\.4%\-7\.4\\%−5\.8%\-5\.8\\%\+78\.4%\\mathbf\{\+78\.4\\%\}\+76\.4%\\mathbf\{\+76\.4\\%\}\+72\.8%\\mathbf\{\+72\.8\\%\}FairTFM\-25−10\.7%\-10\.7\\%−11\.0%\-11\.0\\%\+88\.7%\\mathbf\{\+88\.7\\%\}\+85\.8%\\mathbf\{\+85\.8\\%\}\+83\.7%\\mathbf\{\+83\.7\\%\}
\(b\)vs\. XGBModelAccAUCROCDPEODEOPFairTFM\-0\.1−3\.5%\-3\.5\\%−2\.6%\-2\.6\\%\+48\.0%\\mathbf\{\+48\.0\\%\}\+39\.4%\\mathbf\{\+39\.4\\%\}\+29\.4%\\mathbf\{\+29\.4\\%\}FairTFM\-0\.7−3\.6%\-3\.6\\%±0\.0%\\pm 0\.0\\%\+51\.0%\\mathbf\{\+51\.0\\%\}\+43\.4%\\mathbf\{\+43\.4\\%\}\+34\.1%\\mathbf\{\+34\.1\\%\}FairTFM\-10−5\.9%\-5\.9\\%−2\.9%\-2\.9\\%\+79\.0%\\mathbf\{\+79\.0\\%\}\+74\.7%\\mathbf\{\+74\.7\\%\}\+70\.6%\\mathbf\{\+70\.6\\%\}FairTFM\-25−9\.2%\-9\.2\\%−8\.2%\-8\.2\\%\+89\.0%\\mathbf\{\+89\.0\\%\}\+84\.8%\\mathbf\{\+84\.8\\%\}\+82\.4%\\mathbf\{\+82\.4\\%\}
\(c\)vs\. LRModelAccAUCROCDPEODEOPFairTFM\-0\.1−4\.0%\-4\.0\\%−2\.6%\-2\.6\\%\+47\.5%\\mathbf\{\+47\.5\\%\}\+47\.8%\\mathbf\{\+47\.8\\%\}\+31\.0%\\mathbf\{\+31\.0\\%\}FairTFM\-0\.7−4\.1%\-4\.1\\%±0\.0%\\pm 0\.0\\%\+50\.5%\\mathbf\{\+50\.5\\%\}\+51\.3%\\mathbf\{\+51\.3\\%\}\+35\.6%\\mathbf\{\+35\.6\\%\}FairTFM\-10−6\.4%\-6\.4\\%−2\.9%\-2\.9\\%\+78\.8%\\mathbf\{\+78\.8\\%\}\+78\.3%\\mathbf\{\+78\.3\\%\}\+71\.3%\\mathbf\{\+71\.3\\%\}FairTFM\-25−9\.7%\-9\.7\\%−8\.2%\-8\.2\\%\+88\.9%\\mathbf\{\+88\.9\\%\}\+87\.0%\\mathbf\{\+87\.0\\%\}\+82\.8%\\mathbf\{\+82\.8\\%\}

### E\.7Fairness\-aware baselines on the 12 non\-ACS tasks

We next compareFairTFMagainst task\-specific fairness\-aware baselines on the same 12 non\-ACS tasks\. We wrap XGBoost with the Exponentiated Gradient \(EG\) reduction\[[1](https://arxiv.org/html/2608.14211#bib.bib26)\]; we focus on XGBoost because it is among the strongest non\-TFM baselines on these tasks \(Table[4](https://arxiv.org/html/2608.14211#A5.T4)\) and was the strongest EG\-wrapped model in the ACS comparison \(Figure[5](https://arxiv.org/html/2608.14211#S4.F5)\)\. For each of the three fairness constraints \(DP, EOD, EOP\), we sweep the fairness\-violation toleranceε\\varepsilonover\{0\.01,0\.02,0\.03,0\.05,0\.07,0\.10,0\.20,0\.30,0\.50,0\.70,0\.90\}\\\{0\.01,0\.02,0\.03,0\.05,0\.07,0\.10,0\.20,0\.30,0\.50,0\.70,0\.90\\\}, yielding 33 configurations, each retrained per task and averaged over three seeds\. Table[7](https://arxiv.org/html/2608.14211#A5.T7)reports all configurations, and Table[6](https://arxiv.org/html/2608.14211#A5.T6)summarizes the comparison using, for each constraint type, the best \(lowest\) value of its targeted metric acrossε\\varepsilon\.

Three observations follow\. First, eachFairTFMvariant improves on the fairness metric that each EG variant explicitly optimizes: relative to the best EG value of each targeted metric \(DP 0\.062, EOD 0\.083, EOP 0\.070\),FairTFM\-0\.7 improves by 21% \(DP\), 33% \(EOD\), and 20% \(EOP\),FairTFM\-10 by 66%, 70%, and 64%, andFairTFM\-25 by 82%, 82%, and 79%\. Second, this comes at substantially higher predictive performance: the EG randomized ensembles reduce XGBoost’s AUCROC from 0\.803 to roughly 0\.69–0\.70 regardless ofε\\varepsilon, whileFairTFM\-0\.7 keeps AUCROC at 0\.803 with only a modest accuracy cost\. Third, the comparison highlights a structural advantage: obtaining the EG numbers required retraining33×12×333\\times 12\\times 3model configurations, whereas a single pretrainedFairTFMcheckpoint serves every task and constraint in one forward pass\. This per\-task, per\-constraint retraining cost is itself part of the case for fairness\-aware pretraining\.

Table 6:Summary comparison on the 12 non\-ACS tasks\. For each EG constraint type, we report the configuration achieving the best \(lowest\) value of its targeted metric across theε\\varepsilonsweep \(targeted metric in bold\); unconstrained XGBoost is included as a reference\. EachFairTFMvariant is fairer on every targeted metric at substantially higher AUCROC\.ModelAccuracyAUCROCDP DiffEOD DiffEOP DiffXGB \(unconstrained\)0\.7800\.7800\.8030\.8030\.1000\.1000\.0990\.0990\.0850\.085XGB\-EG\-DP \(bestε\\varepsilon\)0\.7770\.7770\.6950\.6950\.062\\mathbf\{0\.062\}0\.0750\.0750\.0600\.060XGB\-EG\-EOD \(bestε\\varepsilon\)0\.7780\.7780\.7020\.7020\.0840\.0840\.083\\mathbf\{0\.083\}0\.0680\.068XGB\-EG\-EOP \(bestε\\varepsilon\)0\.7770\.7770\.7000\.7000\.0850\.0850\.0840\.0840\.070\\mathbf\{0\.070\}FairTFM\-0\.70\.7520\.7520\.8030\.8030\.0490\.0490\.0560\.0560\.0560\.056FairTFM\-100\.7340\.7340\.7800\.7800\.0210\.0210\.0250\.0250\.0250\.025FairTFM\-250\.7080\.7080\.7370\.7370\.0110\.0110\.0150\.0150\.0150\.015Table 7:Full Exponentiated Gradient results on the 12 non\-ACS tasks: average fairness and accuracy metrics for each fairness constraint \(DP, EOD, EOP\) and toleranceε\\varepsilon, mean±\\pmstandard deviation over three random seeds\.Model𝜺\\boldsymbol\{\\varepsilon\}AccuracyAUCROCDP DiffEOD DiffEOP DiffXGB\-EG\-DP0\.010\.778±\\pm0\.100\.695±\\pm0\.070\.068±\\pm0\.060\.073±\\pm0\.030\.061±\\pm0\.030\.020\.778±\\pm0\.100\.695±\\pm0\.070\.064±\\pm0\.050\.065±\\pm0\.030\.054±\\pm0\.030\.030\.778±\\pm0\.100\.694±\\pm0\.070\.064±\\pm0\.050\.069±\\pm0\.030\.058±\\pm0\.030\.050\.778±\\pm0\.100\.692±\\pm0\.060\.067±\\pm0\.050\.070±\\pm0\.030\.059±\\pm0\.030\.070\.777±\\pm0\.100\.694±\\pm0\.060\.062±\\pm0\.050\.072±\\pm0\.030\.062±\\pm0\.030\.100\.777±\\pm0\.100\.692±\\pm0\.060\.068±\\pm0\.050\.071±\\pm0\.020\.059±\\pm0\.030\.200\.778±\\pm0\.100\.693±\\pm0\.060\.067±\\pm0\.050\.075±\\pm0\.040\.061±\\pm0\.040\.300\.776±\\pm0\.100\.693±\\pm0\.070\.068±\\pm0\.060\.078±\\pm0\.030\.065±\\pm0\.030\.500\.777±\\pm0\.100\.695±\\pm0\.070\.062±\\pm0\.050\.075±\\pm0\.040\.060±\\pm0\.040\.700\.778±\\pm0\.100\.696±\\pm0\.070\.072±\\pm0\.060\.083±\\pm0\.040\.068±\\pm0\.040\.900\.777±\\pm0\.100\.695±\\pm0\.060\.066±\\pm0\.060\.071±\\pm0\.040\.059±\\pm0\.04XGB\-EG\-EOD0\.010\.778±\\pm0\.100\.702±\\pm0\.070\.084±\\pm0\.080\.084±\\pm0\.050\.069±\\pm0\.050\.020\.778±\\pm0\.100\.702±\\pm0\.070\.084±\\pm0\.080\.083±\\pm0\.050\.068±\\pm0\.050\.030\.778±\\pm0\.100\.704±\\pm0\.070\.084±\\pm0\.080\.086±\\pm0\.050\.071±\\pm0\.050\.050\.778±\\pm0\.100\.704±\\pm0\.070\.086±\\pm0\.080\.090±\\pm0\.050\.075±\\pm0\.050\.070\.778±\\pm0\.100\.703±\\pm0\.070\.086±\\pm0\.080\.088±\\pm0\.050\.074±\\pm0\.050\.100\.778±\\pm0\.100\.703±\\pm0\.070\.085±\\pm0\.080\.084±\\pm0\.050\.067±\\pm0\.050\.200\.778±\\pm0\.100\.700±\\pm0\.070\.086±\\pm0\.080\.087±\\pm0\.060\.072±\\pm0\.060\.300\.779±\\pm0\.100\.700±\\pm0\.070\.085±\\pm0\.070\.084±\\pm0\.060\.069±\\pm0\.050\.500\.777±\\pm0\.100\.701±\\pm0\.070\.084±\\pm0\.080\.089±\\pm0\.060\.076±\\pm0\.060\.700\.777±\\pm0\.100\.700±\\pm0\.070\.086±\\pm0\.080\.087±\\pm0\.060\.074±\\pm0\.060\.900\.778±\\pm0\.100\.701±\\pm0\.070\.088±\\pm0\.080\.088±\\pm0\.060\.074±\\pm0\.06XGB\-EG\-EOP0\.010\.776±\\pm0\.100\.698±\\pm0\.070\.087±\\pm0\.080\.089±\\pm0\.060\.074±\\pm0\.060\.020\.776±\\pm0\.100\.698±\\pm0\.070\.085±\\pm0\.080\.087±\\pm0\.060\.072±\\pm0\.060\.030\.776±\\pm0\.100\.699±\\pm0\.070\.086±\\pm0\.080\.088±\\pm0\.060\.072±\\pm0\.060\.050\.777±\\pm0\.100\.699±\\pm0\.070\.084±\\pm0\.080\.087±\\pm0\.060\.073±\\pm0\.060\.070\.776±\\pm0\.100\.699±\\pm0\.070\.085±\\pm0\.080\.086±\\pm0\.060\.073±\\pm0\.060\.100\.777±\\pm0\.100\.699±\\pm0\.070\.087±\\pm0\.080\.087±\\pm0\.060\.074±\\pm0\.060\.200\.778±\\pm0\.100\.699±\\pm0\.070\.086±\\pm0\.080\.085±\\pm0\.060\.071±\\pm0\.060\.300\.777±\\pm0\.100\.700±\\pm0\.070\.085±\\pm0\.090\.084±\\pm0\.060\.070±\\pm0\.060\.500\.777±\\pm0\.100\.700±\\pm0\.070\.088±\\pm0\.080\.088±\\pm0\.060\.075±\\pm0\.060\.700\.778±\\pm0\.100\.699±\\pm0\.070\.091±\\pm0\.090\.092±\\pm0\.060\.077±\\pm0\.060\.900\.777±\\pm0\.100\.701±\\pm0\.070\.086±\\pm0\.080\.091±\\pm0\.060\.080±\\pm0\.06