TabLoRA: Parameter-Efficient Low-Rank Ensemble Learning for Large-Scale Tabular Data

# TabLoRA: Parameter-Efficient Low-Rank Ensemble Learning for Large-Scale Tabular Data
Source: [https://arxiv.org/html/2607.10077](https://arxiv.org/html/2607.10077)
###### Abstract

Tabular learning is still dominated by gradient\-boosted decision trees \(GBDTs\), while recent deep learning approaches have become increasingly competitive\. However, applying deep tabular models to large\-scale datasets remains challenging, as large sample sizes, high feature dimensionality, or many target classes can introduce substantial computational cost\. We proposeTabLoRA, a parameter\-efficient trainable neural ensemble for large\-scale tabular learning\. Instead of using fully independent ensemble backbones, TabLoRA shares a common backbone across predictors and introduces predictor\-specific low\-rank adaptations, enabling ensemble\-style prediction without full parameter duplication\. Across benchmarks, TabLoRA achieves a favorable balance between predictive performance and practical efficiency compared with GBDT methods and recent deep learning baselines under the same resource constraints\. Memory analysis and ablation studies further show that the proposed design improves the feasibility of neural ensemble learning while preserving much of the benefit of full ensembles\.

###### keywords:

Large\-scale Tabular Data , Deep Learning , Low\-Rank Adaptation , Ensemble , Parameter\-Efficient

††journal:arXiv\\affiliation

\[first\]organization=School of Mathematical Sciences, Soochow University, addressline=No\.1 Shizi Street, city=Suzhou, postcode=215006, state=Jiangsu Province, country=China

\\affiliation

\[second\]organization=Digital innovation research center, Duke Kunshan University,addressline=No\.8 Duke Avenue, city=Kunshan, postcode=215000, state=Jiangsu Province, country=China

## 1Introduction

Tabular data remains a fundamental modality in real\-world machine learning applications\[[25](https://arxiv.org/html/2607.10077#bib.bib102)\]\. Historically, gradient\-boosted decision trees \(GBDTs\)\[[7](https://arxiv.org/html/2607.10077#bib.bib1),[27](https://arxiv.org/html/2607.10077#bib.bib11),[33](https://arxiv.org/html/2607.10077#bib.bib7)\]have long dominated this domain due to their strong empirical performance and robustness\[[3](https://arxiv.org/html/2607.10077#bib.bib14),[37](https://arxiv.org/html/2607.10077#bib.bib34),[16](https://arxiv.org/html/2607.10077#bib.bib20)\]\. Recently, however, tabular deep learning has become increasingly competitive\. In particular, in\-context learning methods such as TabPFN\[[19](https://arxiv.org/html/2607.10077#bib.bib54),[20](https://arxiv.org/html/2607.10077#bib.bib74),[15](https://arxiv.org/html/2607.10077#bib.bib76)\]demonstrate that deep learning models can match or even outperform GBDT on small\- and medium\-scale datasets when properly formulated\.

Despite this progress, large\-scale tabular learning remains computationally challenging\. Here, large\-scale datasets refer to settings where scale may arise from a large number of samples, high feature dimensionality, or many target classes\. In these regimes, GBDTs become bottlenecked by tree construction, split search, and class\-wise modeling costs, while deep learning models suffer from prohibitive memory and computational footprints during training, making these methods costly or even infeasible\.

Among deep learning approaches, MLP\-based models provide a simple and computationally efficient foundation, making them naturally amenable to large\-scale settings\. Recent work further shows that incorporating classical machine learning structures\[[30](https://arxiv.org/html/2607.10077#bib.bib69),[10](https://arxiv.org/html/2607.10077#bib.bib72),[45](https://arxiv.org/html/2607.10077#bib.bib71)\]can significantly enhance their performance\. Compared to more complex architectures, such designs offer practical efficiency and are easier to scale to large datasets\.

However, even within this design space, a fundamental challenge remains\. Highly expressive MLP\-based models often incur substantial memory overhead that grows with feature dimensionality and dataset size, while memory\-efficient designs tend to suffer from limited representation capacity\. This reveals an inherent trade\-off between scalability and expressivity in large\-scale tabular learning, which remains insufficiently addressed by existing approaches\.

To overcome this limitation, we proposeTabLoRA, a parameter\-efficient ensemble framework that combines shared backbone learning with predictor\-specific low\-rank adaptations\. Each predictor is modeled as a low\-rank perturbation of shared weights, allowing the model to approximate the behavior of deep ensembles without incurring linear growth in memory cost\. In addition, we introduce lightweight feature transformations that generate multiple input representations, further enhancing diversity across predictors without increasing feature dimensionality\. Experimental results on large\-scale dataset benchmarks show that TabLoRA achieves strong predictive performance and a favorable practical performance–efficiency trade\-off compared with the state\-of\-the\-art baselines\. Further ablation studies demonstrate that TabLoRA significantly reduces the parameters of neural ensemble learning while preserving the predictive capability of full ensembles\.

The main contributions of this paper are summarized as follows:

- 1\.We propose TabLoRA, a parameter\-efficient trainable neural ensemble for large\-scale tabular learning\.
- 2\.We design a shared\-backbone, low\-rank adaptation mechanism that enables predictor\-specific specialization without full ensemble parameter duplication\.
- 3\.We empirically show that TabLoRA achieves a favorable performance–efficiency trade\-off on large\-scale tabular benchmarks, with ablations validating the role of the feature adapter and low\-rank ensemble parameterization\.

## 2Related Work

### 2\.1Deep Learning for Tabular Data

This section gives an overview of relevant concepts from prior research on deep learning for tabular data\. We categorize deep learning models into three types based on their network structure: tree\-induced networks, transformer\-based networks, and other specialized models\.

##### MLP\-based Architectures

MLP\-based models form one of the most practical directions in tabular deep learning\. Although plain MLPs often underperform GBDT methods, recent studies show that their performance can be substantially improved by incorporating techniques from classical machine learning, better regularization, feature encoding, retrieval, and ensemble learning\. Early efforts improve the training and generalization of MLPs through regularization and carefully designed default settings\[[36](https://arxiv.org/html/2607.10077#bib.bib33),[26](https://arxiv.org/html/2607.10077#bib.bib94),[21](https://arxiv.org/html/2607.10077#bib.bib75)\]\. Another line of work focuses on improving input representations\. For example, MLP\-PLR\[[11](https://arxiv.org/html/2607.10077#bib.bib95)\]introduces numerical feature encoding methods to better represent continuous variables, while TabR\[[12](https://arxiv.org/html/2607.10077#bib.bib92)\]augments MLP\-based prediction with retrieval mechanisms to improve robustness and predictive performance\. Other methods integrate classical learning principles more directly into neural architectures\. ModernNCA\[[45](https://arxiv.org/html/2607.10077#bib.bib71)\]incorporates Neighbourhood Component Analysis into an MLP\-based framework, whereas TabM\[[10](https://arxiv.org/html/2607.10077#bib.bib72)\]uses MLPs as base learners and adopts BatchEnsemble and numerical feature encoding to construct multiple diverse predictors efficiently\. Similarly, NCART\[[30](https://arxiv.org/html/2607.10077#bib.bib69)\]combines MLPs with decision\-tree\-based ensemble learning\. Together, these studies suggest that MLP\-like backbones remain competitive when equipped with suitable tabular\-specific mechanisms\.

##### Transformer\-based Architectures

Transformer\-based models provide another important direction for tabular learning by explicitly modeling interactions among features or samples through attention mechanisms\. Inspired by the success of Transformers\[[41](https://arxiv.org/html/2607.10077#bib.bib13)\], early methods adapt self\-attention to heterogeneous tabular inputs\. TabTransformer\[[24](https://arxiv.org/html/2607.10077#bib.bib23)\]maps categorical features into contextual embeddings using self\-attention, improving robustness to missing or noisy values and offering a degree of interpretability\. FT\-Transformer\[[13](https://arxiv.org/html/2607.10077#bib.bib19)\]provides a more direct adaptation by tokenizing both numerical and categorical features and feeding them jointly into Transformer blocks\. Beyond feature\-wise attention, several methods introduce more structured attention mechanisms\. TabNet\[[2](https://arxiv.org/html/2607.10077#bib.bib9)\]adopts a sequential decision process with soft instance\-wise feature selection, while SAINT\[[38](https://arxiv.org/html/2607.10077#bib.bib29)\]combines feature\-wise self\-attention with inter\-sample attention and further uses self\-supervised contrastive pre\-training\. NPT\[[28](https://arxiv.org/html/2607.10077#bib.bib30)\]treats the entire dataset as input and uses attention between data points to model sample\-level relationships\. More recent models further refine Transformer\-based tabular learning with specialized interaction modules or structural information\. ExcelFormer\[[6](https://arxiv.org/html/2607.10077#bib.bib93)\]alternates between attention modules for feature interaction and embedding updates, and T2GFormer\[[44](https://arxiv.org/html/2607.10077#bib.bib60)\]explores Transformer\-based modeling with graph\-based structures\. These methods demonstrate the flexibility of attention mechanisms for tabular data, although their computational cost can become a concern in large\-scale settings\.

##### Other Specific Architectures

Beyond MLP\- and Transformer\-based models, many studies design specialized architectures to address the structural heterogeneity of tabular data\. Unlike images or language, tabular data does not have a universal spatial or sequential structure, which makes it difficult to directly transfer standard deep learning architectures\. One strategy is to transform tabular data into other modalities, such as images or text, and then apply models designed for those domains\[[39](https://arxiv.org/html/2607.10077#bib.bib45),[46](https://arxiv.org/html/2607.10077#bib.bib44),[18](https://arxiv.org/html/2607.10077#bib.bib51)\]\. Another strategy is to design task\-specific neural architectures that better capture feature interactions and high\-level representations\. For instance, DANets\[[5](https://arxiv.org/html/2607.10077#bib.bib62)\]and TabCaps\[[4](https://arxiv.org/html/2607.10077#bib.bib58)\]introduce specialized structures for representation learning, while NODE\[[32](https://arxiv.org/html/2607.10077#bib.bib10)\]integrates differentiable oblivious decision trees into an ensemble\-style neural model\. In addition, self\-supervised learning methods\[[47](https://arxiv.org/html/2607.10077#bib.bib46),[40](https://arxiv.org/html/2607.10077#bib.bib64),[17](https://arxiv.org/html/2607.10077#bib.bib68)\]and transfer learning approaches\[[43](https://arxiv.org/html/2607.10077#bib.bib57),[29](https://arxiv.org/html/2607.10077#bib.bib67)\]have been introduced to improve representation learning and downstream predictive performance\. These works reflect the diversity of architectural designs for tabular deep learning\.

##### Pre\-trained Models

Pre\-trained tabular models have recently emerged as a promising direction, aiming to transfer knowledge across datasets and adapt to new tabular tasks with limited task\-specific training\. A representative example is TabPFN\[[19](https://arxiv.org/html/2607.10077#bib.bib54),[20](https://arxiv.org/html/2607.10077#bib.bib74)\], which demonstrates the effectiveness of in\-context learning for small\-scale tabular prediction and has recently been extended toward more scalable settings\[[14](https://arxiv.org/html/2607.10077#bib.bib73),[15](https://arxiv.org/html/2607.10077#bib.bib76)\]\. Following this direction, several works further improve scalability, generality, or task coverage\. TabDPT\[[31](https://arxiv.org/html/2607.10077#bib.bib98)\]combines retrieval techniques with self\-supervised learning to train tabular foundation models, while TabICL\[[34](https://arxiv.org/html/2607.10077#bib.bib96),[35](https://arxiv.org/html/2607.10077#bib.bib97)\]employs a column\-then\-row attention mechanism to address the scalability limitations of TabPFN\. LimiX\[[49](https://arxiv.org/html/2607.10077#bib.bib99),[42](https://arxiv.org/html/2607.10077#bib.bib100)\]further broadens the scope by using a single model for multiple tabular tasks, including classification, regression, missing\-value imputation, feature selection, and sample selection\. These methods show the potential of pre\-trained tabular predictors, although their inference and adaptation costs remain important considerations in large\-scale applications\.

### 2\.2Parameter\-Efficient Fine\-Tuning

Parameter\-efficient fine\-tuning \(PEFT\)\[[9](https://arxiv.org/html/2607.10077#bib.bib77)\]aims to adapt pretrained models to downstream tasks by updating only a small number of task\-specific parameters\. Instead of fine\-tuning the entire backbone, PEFT methods usually freeze most pretrained weights and introduce lightweight trainable components, such as adapters and low\-rank updates\.

Adapter tuning is a representative PEFT strategy\. Houlsby et al\.\[[22](https://arxiv.org/html/2607.10077#bib.bib83)\]introduced bottleneck adapters, where small trainable modules are inserted into pretrained networks while the backbone remains fixed\. Another important PEFT method is low\-rank adaptation\. LoRA\[[23](https://arxiv.org/html/2607.10077#bib.bib81)\]freezes pretrained weights and injects trainable low\-rank matrices into selected layers, representing weight updates through low\-rank decompositions\. This greatly reduces the number of trainable parameters while maintaining competitive performance\. Several extensions further improve LoRA’s flexibility and efficiency\. AdaLoRA\[[48](https://arxiv.org/html/2607.10077#bib.bib86)\]dynamically allocates the parameter budget across weight matrices, whereas QLoRA\[[8](https://arxiv.org/html/2607.10077#bib.bib87)\]combines LoRA with three innovations to enable memory\-efficient fine\-tuning of large language models\.

## 3Methodology

### 3\.1Overview

![Refer to caption](https://arxiv.org/html/2607.10077v1/x1.png)Figure 1:Overview of TabLoRA\. The model combines predictor\-specific feature adaptation, shared backbone learning with low\-rank adaptations, and ensemble aggregation\.We propose a parameter\-efficient ensemble framework for tabular learning\. As illustrated in Fig\.[1](https://arxiv.org/html/2607.10077#S3.F1), our method constructs multiple predictors through a shared backbone with low\-rank adaptations, enabling efficient ensemble modeling\.

Given an input samplex∈ℝDx\\in\\mathbb\{R\}^\{D\}, the model producesKKpredictions\{fk​\(x\)\}k=1K\\\{f\_\{k\}\(x\)\\\}\_\{k=1\}^\{K\}in parallel, which are later aggregated during training and inference\. The overall architecture consists of three components: \(1\) a feature adapter that generatesKKrepresentations, \(2\) a shared backbone with low\-rank adaptations, and \(3\) a group\-wise prediction head\.

### 3\.2Feature Expansion via Group Adapter

To construct multiple predictors efficiently, we first transform the input feature vector intoKKparallel representations\. Given an input featurex∈ℝDx\\in\\mathbb\{R\}^\{D\}, we define

xk=x\+ϕk​\(x\),x\_\{k\}=x\+\\phi\_\{k\}\(x\),\(1\)whereϕk​\(⋅\)\\phi\_\{k\}\(\\cdot\)is an adapter\[[22](https://arxiv.org/html/2607.10077#bib.bib83)\]:

ϕk​\(x\)=Wup\(k\)​σ​\(Wdown\(k\)​x\)\.\\phi\_\{k\}\(x\)=W^\{\(k\)\}\_\{\\text\{up\}\}\\,\\sigma\\left\(W^\{\(k\)\}\_\{\\text\{down\}\}x\\right\)\.\(2\)Here,Wup\(k\)∈ℝD×rW^\{\(k\)\}\_\{\\text\{up\}\}\\in\\mathbb\{R\}^\{D\\times r\}andWdown\(k\)∈ℝr×DW^\{\(k\)\}\_\{\\text\{down\}\}\\in\\mathbb\{R\}^\{r\\times D\}are trainable,σ​\(⋅\)\\sigma\(\\cdot\)is a non\-linear activation\. This results inKKfeature representations

X=\[x1,x2,…,xK\]∈ℝK×D\.X=\[x\_\{1\},x\_\{2\},\\dots,x\_\{K\}\]\\in\\mathbb\{R\}^\{K\\times D\}\.\(3\)This design introduces diversity across predictors while maintaining a small parameter footprint due to the low\-rank bottleneck\.

### 3\.3Parameter\-Efficient Ensemble via Low\-Rank Adaptation

Instead of trainingKKindependent networks, we use a shared backbone with per\-predictor low\-rank adaptations\[[23](https://arxiv.org/html/2607.10077#bib.bib81)\]\. Unlike conventional LoRA, which freezes a pretrained backbone for parameter\-efficient fine\-tuning, TabLoRA uses low\-rank adaptation as an ensemble parameterization\. Specifically, the shared backbone and the predictor\-specific low\-rank adaptations are trained jointly from scratch for tabular ensemble learning\. For a linear layer with weightW∈ℝdout×dinW\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times d\_\{\\text\{in\}\}\}, the output for predictorkkis defined as

hk=W​xk\+αr⋅Bk​Ak​xk,k=1,2,⋯,K,h\_\{k\}=Wx\_\{k\}\+\\frac\{\\alpha\}\{r\}\\cdot B\_\{k\}A\_\{k\}x\_\{k\},k=1,2,\\cdots,K,\(4\)whereAk∈ℝr×dinA\_\{k\}\\in\\mathbb\{R\}^\{r\\times d\_\{\\text\{in\}\}\},Bk∈ℝdout×rB\_\{k\}\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times r\},r≪din,doutr\\ll d\_\{\\text\{in\}\},d\_\{\\text\{out\}\}is the low\-rank dimension, andα\\alphais a scaling factor\.

The first term corresponds to the shared backbone, while the second term introduces predictor\-specific low\-rank perturbations\. This formulation enables the model to approximate multiple predictors with significantly fewer parameters than independent ensembles\.

The network is implemented as a multilayer MLP\. All linear layers in the network are replaced with the low\-rank ensemble backbones described above\. Given theKKfeature representationsxklx\_\{k\}^\{l\}of the layerll, the backbone processes them in parallel

xkl\+1=R​e​L​U​\(Wl​xkl\+αr⋅Bkl​Akl​xkl\+bl\),k=1,2,⋯,K,x\_\{k\}^\{l\+1\}=ReLU\\left\(W^\{l\}x\_\{k\}^\{l\}\+\\frac\{\\alpha\}\{r\}\\cdot B\_\{k\}^\{l\}A\_\{k\}^\{l\}x\_\{k\}^\{l\}\+b^\{l\}\\right\),k=1,2,\\cdots,K,\(5\)whereblb\_\{l\}is the shared bias andR​e​L​UReLUis the relu activative function\.

### 3\.4Group Prediction Head

To generate predictions for all ensemble members, we use a group\-wise linear head

zk=Wkh​e​a​d​xkL\+bkh​e​a​d,z\_\{k\}=W\_\{k\}^\{head\}x\_\{k\}^\{L\}\+b\_\{k\}^\{head\},\(6\)where each predictor has its own head parametersWkh​e​a​dW\_\{k\}^\{head\}andbkh​e​a​db\_\{k\}^\{head\}\. The final output is

Z=\[z1,z2,…,zK\]\.Z=\[z\_\{1\},z\_\{2\},\\dots,z\_\{K\}\]\.\(7\)

### 3\.5Training and Inference

During training, all predictors are treated equally by optimizing the average loss over theKKoutputs:

ℒtotal=1K​∑k=1Kℒ​\(zk,y\),\\mathcal\{L\}\_\{\\mathrm\{total\}\}=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathcal\{L\}\(z\_\{k\},y\),\(8\)whereyyis the ground\-truth label\. For classification,ℒ\\mathcal\{L\}is the cross\-entropy loss applied directly to the logitszkz\_\{k\}\. For regression,ℒ\\mathcal\{L\}is a regression loss such as mean squared error\.

During inference, theKKpredictor outputs are aggregated by averaging\. For regression, the final prediction is

y^=1K​∑k=1Kzk\.\\hat\{y\}=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}z\_\{k\}\.\(9\)For classification, we aggregate the logits and then apply the softmax function:

p^=softmax​\(1K​∑k=1Kzk\),\\hat\{p\}=\\mathrm\{softmax\}\\left\(\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}z\_\{k\}\\right\),\(10\)wherep^\\hat\{p\}is the predicted class probability vector\.

### 3\.6Complexity Analysis

We compare the parameter complexity of TabLoRA with a standard deep ensemble\. For simplicity, we focus on the backbone parameters and omit lower\-order terms such as biases and prediction heads\. Consider anLL\-layer MLP backbone with input dimensionddand hidden dimensionhh\. A standard deep ensemble withKKindependent predictors duplicates the entire backbone for each predictor, resulting in

Pfull=K​\(d​h\+\(L−1\)​h2\)\.P\_\{\\mathrm\{full\}\}=K\\big\(dh\+\(L\-1\)h^\{2\}\\big\)\.\(11\)
In contrast, TabLoRA shares the main backbone across all predictors\. The shared backbone contains

Pshared=d​h\+\(L−1\)​h2P\_\{\\mathrm\{shared\}\}=dh\+\(L\-1\)h^\{2\}\(12\)parameters\. For each predictor, TabLoRA introduces low\-rank adaptations to the backbone weights\. For the input layer, whose weight matrix has sizeh×dh\\times d, the low\-rank adaptation introducesr​\(d\+h\)r\(d\+h\)parameters\. For each hidden layer, whose weight matrix has sizeh×hh\\times h, the low\-rank adaptation introduces2​h​r2hrparameters\. Therefore, the predictor\-specific low\-rank parameters for allKKpredictors are

PLoRA=K​\(r​\(d\+h\)\+2​\(L−1\)​h​r\)=K​r​\(d\+\(2​L−1\)​h\),P\_\{\\mathrm\{LoRA\}\}=K\\Big\(r\(d\+h\)\+2\(L\-1\)hr\\Big\)=Kr\\big\(d\+\(2L\-1\)h\\big\),\(13\)wherer≪hr\\ll his the adaptation rank\.

The total backbone\-related parameter complexity of TabLoRA is therefore

PTabLoRA=d​h\+\(L−1\)​h2\+K​r​\(d\+\(2​L−1\)​h\)\.P\_\{\\mathrm\{TabLoRA\}\}=dh\+\(L\-1\)h^\{2\}\+Kr\\big\(d\+\(2L\-1\)h\\big\)\.\(14\)
Compared with the standard ensemble, TabLoRA avoids duplicating the full backbone for each predictor\. The ensemble\-specific parameter growth is reduced fromK​h​\(d\+\(L−1\)​h\)Kh\\big\(d\+\(L\-1\)h\\big\)toK​r​\(d\+\(2​L−1\)​h\)Kr\\big\(d\+\(2L\-1\)h\\big\)\. Sincer≪hr\\ll h, the predictor\-specific cost in TabLoRA is much smaller than maintainingKKindependent backbones\. Thus, TabLoRA provides a parameter\-efficient ensemble parameterization while preserving multiple predictor\-specific adaptations\.

## 4Experimental Setup

### 4\.1Datasets and Baselines

We conduct experiments on 16 tabular datasets from OpenML111[https://www\.openml\.org/](https://www.openml.org/), covering both classification and regression tasks\. The classification datasets include 5 binary and 7 multi\-class problems\. Dataset details can be found in[A](https://arxiv.org/html/2607.10077#A1)\. We compare the proposed method with three widely used GBDT models and five representative deep tabular learning baselines\. The GBDT baselines include XGBoost\[[7](https://arxiv.org/html/2607.10077#bib.bib1)\], CatBoost\[[33](https://arxiv.org/html/2607.10077#bib.bib7)\], and LightGBM\[[27](https://arxiv.org/html/2607.10077#bib.bib11)\]\. The neural baselines include MLP, RealMLP\[[21](https://arxiv.org/html/2607.10077#bib.bib75)\], NCART\[[30](https://arxiv.org/html/2607.10077#bib.bib69)\], TabM\[[10](https://arxiv.org/html/2607.10077#bib.bib72)\], and TabPFN\[[15](https://arxiv.org/html/2607.10077#bib.bib76)\]\.

### 4\.2Evaluation Metrics

We evaluate model performance usingAUC\(Area Under the Curve\) for binary classification andAcc\.\(Accuracy\) for multi\-class classification\. For regression tasks, we useMSE\(Mean Squared Error\)\. Since these metrics have different directions and scales, we define a unified relative improvement metric based on gap reduction with respect to MLP\. For each datasetddand methodmm, we define the relative improvement as

Im,d=\{\(1−SMLP,d\)−\(1−Sm,d\)1−SMLP,d,classification,MSEMLP,d−MSEm,dMSEMLP,d,regression,I\_\{m,d\}=\\begin\{cases\}\\dfrac\{\(1\-S\_\{\\mathrm\{MLP\},d\}\)\-\(1\-S\_\{m,d\}\)\}\{1\-S\_\{\\mathrm\{MLP\},d\}\},&\\text\{classification\},\\\\\[11\.99998pt\] \\dfrac\{\\mathrm\{MSE\}\_\{\\mathrm\{MLP\},d\}\-\\mathrm\{MSE\}\_\{m,d\}\}\{\\mathrm\{MSE\}\_\{\\mathrm\{MLP\},d\}\},&\\text\{regression\},\\end\{cases\}\(15\)whereSm,dS\_\{m,d\}denotes AUC for binary classification and Acc\. for multi\-class classification\. A positive value ofIm,dI\_\{m,d\}indicates that methodmmimproves over MLP, while a negative value indicates worse performance\.

### 4\.3Implementation Details

To ensure fair and reproducible evaluation, we adopt a standardized experimental protocol across all datasets and methods\. Each dataset is randomly split into training 80% and testing 20% sets using stratified sampling to preserve class distribution\. Within the training set, 10% is further reserved as a validation set for hyperparameter tuning\. Hyperparameters are optimized using Optuna\[[1](https://arxiv.org/html/2607.10077#bib.bib18)\]with a budget of 10 trials per method\. After tuning, models are retrained on the full training set and evaluated on the test set\. To account for randomness, we repeat the entire process five times with different random seeds and report the mean and standard deviation of all metrics\.

For neural network models, we train for up to 200 epochs using the Adam optimizer with a learning rate of 0\.001\. For GBDT models, the number of boosting iterations is set to 200\. Early stopping is applied with a patience of 20 epochs or iterations\. The training batch size is set to 1024, and the validation batch size is set to 256\. For each algorithm and each training process, we run the algorithm for up to 10 hours to prevent excessively long runtimes\. All experiments are conducted on a workstation equipped with an Intel Core i9\-14900KF CPU, 128GB RAM, and a 24G NVIDIA\-4090 GPU\. More details about the hyperparameters can be found in[B](https://arxiv.org/html/2607.10077#A2)\.

## 5Results

### 5\.1Main Results on Large\-Scale Datasets

Table[1](https://arxiv.org/html/2607.10077#S5.T1)summarizes the predictive performance on 16 large\-scale datasets\. The results also reveal clear differences in robustness and feasibility across methods\. TabPFN achieves the largest number of best results when it runs successfully, especially on several multi\-class and regression datasets\. However, it suffers from multiple OOM failures on large\-scale datasets, reflecting the computational burden of applying in\-context tabular prediction under large\-scale data regimes\. TabM also obtains strong results on some datasets, but fails on several large\-scale datasets and gives the worst performance on multiple tasks\. This is related to its feature embedding mechanism, which can increase the cost of intermediate representations\. NCART also faces OOM failures on some datasets, since it constructs ensemble\-style neural tree models and can become expensive when the feature dimension is high\. RealMLP improves over the standard MLP with better training strategies and default configurations\. As a single\-network model, it avoids OOM failures but its predictive performance is less competitive than stronger ensemble\-style or foundation\-model baselines\.

GBDT models remain reliable baselines, but their overall ranks are worse than TabLoRA\. XGBoost and CatBoost also encounter OOM failures on some large\-scale datasets\. These failures mainly occur on high\-dimensional datasets, suggesting that the memory cost of tree construction is strongly affected by feature\-wise split statistics and intermediate buffers, rather than by sample size alone\. In contrast, LightGBM does not encounter OOM failures in our experiments, which may be attributed to its histogram\-based learning strategy and memory\-efficient implementation for large\-scale tree construction\.

This observation also indicates that different forms of scale affect different model families differently\. High feature dimensionality is particularly challenging for tree\-based and feature\-embedding methods, while in\-context tabular prediction can also be sensitive to large sample sizes and context construction\. The standard MLP baseline performs poorly overall, confirming that simply scaling a plain neural network is insufficient for large\-scale tabular prediction\. In comparison, TabLoRA obtains the best result on four datasets and is not the worst method on any dataset, showing a strong balance between predictive performance and practical feasibility\.

Fig\.[2](https://arxiv.org/html/2607.10077#S5.F2)summarizes the overall ranking across all large\-scale datasets\. TabLoRA achieves the best average rank among all compared methods, indicating that its advantage is not limited to a few individual datasets but is consistent across different task types\. Fig\.[3](https://arxiv.org/html/2607.10077#S5.F3)further provides a dataset\-level view of relative improvement over MLP\. All TabLoRA points lie on the positive side, showing that TabLoRA consistently improves over the plain MLP baseline\. TabPFN also shows strong positive improvements on successful runs, but its OOM cases limit its practical applicability\. In contrast, several competing methods either show larger variability or suffer from OOM failures on large\-scale datasets\. Together with the main result table, these figures show that TabLoRA achieves strong and stable predictive performance while maintaining good feasibility across the evaluated large\-scale datasets\.

Table 1:Mean±\\pmstandard deviation results of nine models on 16 large\-scale datasets over five random runs\. The best mean result on each dataset is highlighted inbold\. For classification tasks, higher AUC or accuracy is better; for regression tasks, lower MSE is better\.OOMindicates an out\-of\-memory failure under the same GPU budget\.
![Refer to caption](https://arxiv.org/html/2607.10077v1/x2.png)Figure 2:Average performance ranks on large\-scale datasets\. Lower rank indicates better performance\. Blue bars denote deep learning\-based methods, and green bars denote GBDT methods\.![Refer to caption](https://arxiv.org/html/2607.10077v1/x3.png)Figure 3:Dataset\-level relative improvement over MLP on large\-scale datasets\. Each point represents one dataset, and the diamond marker denotes the mean improvement over successful runs\. Cross markers indicateOOMcases, which are shown separately and are not assigned numerical improvement values\.
### 5\.2Practical Efficiency Analysis

We further analyze the practical GPU memory usage of deep learning\-based methods\. For each method, we compute the GPU memory multiple relative to MLP and compare it with the average performance rank across the 16 large\-scale datasets\. Fig\. reff\.bubble and Fig\. reff\.gpu summarize the performance–memory trade\-off and the dataset\-level memory behavior, respectively\. The full numerical results are provided in Table[4](https://arxiv.org/html/2607.10077#A3.T4)in Appendix[C](https://arxiv.org/html/2607.10077#A3)\.

Fig\.[4](https://arxiv.org/html/2607.10077#S5.F4)shows that TabLoRA achieves a favorable balance between predictive performance and average GPU memory usage\. Compared with other neural baselines, TabLoRA obtains the lowest average rank while keeping the mean GPU memory multiple relatively low\. In contrast, TabM and TabPFN require substantially higher average GPU memory and suffer from several OOM failures, although they can be competitive on datasets where they successfully run\. RealMLP is a single\-network model, but its average memory usage is still higher than that of TabLoRA, which may be related to its numerical feature embedding mechanism\. NCART has an average memory usage close to TabLoRA on successful runs, but it also encounters multiple OOM failures, indicating that a low average memory multiple does not necessarily imply stable feasibility across datasets\.

Fig\.[5](https://arxiv.org/html/2607.10077#S5.F5)further shows the dataset\-level GPU memory multiple relative to MLP\. TabLoRA avoids OOM failures across all evaluated datasets and maintains a stable memory footprint\. NCART uses relatively low memory on most successful runs, but fails on several datasets, suggesting that its memory behavior is less stable under large\-scale settings\. TabPFN and TabM show larger memory variation and multiple OOM cases, while RealMLP generally runs successfully but often requires more memory than TabLoRA\. These results indicate that TabLoRA achieves a more reliable performance–memory trade\-off under practical GPU memory constraints\.

![Refer to caption](https://arxiv.org/html/2607.10077v1/x4.png)Figure 4:Performance–memory trade\-off on large\-scale datasets\. The x\-axis reports the mean performance rank, and the y\-axis reports the mean GPU memory multiple relative to MLP\. Lower values on both axes indicate better performance and lower GPU memory usage\. OOM counts are annotated next to the corresponding methods\.![Refer to caption](https://arxiv.org/html/2607.10077v1/x5.png)Figure 5:Dataset\-level GPU memory multiple relative to MLP\. Each point represents one successful dataset\-level run, and the diamond marker denotes the mean multiple over successful runs\. Cross markers indicate OOM cases, which are shown separately and are not assigned numerical memory multiples\.
### 5\.3Ablation Study

We conduct ablation studies to analyze the contributions of two key components in TabLoRA: the feature adapter and the low\-rank ensemble parameterization\. The first ablation evaluates whether the feature adapter improves the input representation\. The second ablation examines whether the low\-rank ensemble parameterization preserves the benefit of full ensembles while reducing parameter growth\. The complete numerical results for all ablation experiments in this subsection are summarized in Table[5](https://arxiv.org/html/2607.10077#A3.T5)in[C](https://arxiv.org/html/2607.10077#A3)\.

#### 5\.3\.1Effect of Feature Adapter

We first study the effect of the feature adapter\. The variant without the feature adapter is referred to asRaw Input, where all ensemble predictors receive the same original input representation\. In contrast, the TabLoRA model uses the feature adapter to generate predictor\-specific input representations before the shared backbone\.

Fig\.[6](https://arxiv.org/html/2607.10077#S5.F6)reports the dataset\-level improvement of TabLoRA over Raw Input\. Positive values indicate that the feature adapter improves performance, while negative values indicate degradation\. The results show that the feature adapter improves performance on most datasets\. Only two datasets show negative improvement, suggesting that the adapter is generally beneficial but still has dataset\-dependent effects\. Large improvements on datasets such as Yolanda, CIFAR\-100, robert, and SVHN suggest that the feature adapter increases ensemble diversity by allowing different predictors to receive different input representations before the shared backbone\.

![Refer to caption](https://arxiv.org/html/2607.10077v1/x6.png)Figure 6:Dataset\-level improvement of TabLoRA over Raw Input\. Positive values indicate that the feature adapter improves performance\. The x\-axis uses a symmetric logarithmic scale to show both small and large improvements\.
#### 5\.3\.2Effect of Low\-Rank Ensemble Parameterization

We next analyze the effect of the low\-rank ensemble parameterization\. We compare TabLoRA with two variants\.Singledenotes a single predictor without ensemble modeling\.Fulldenotes a full deep ensemble, where each ensemble member has an independent backbone\. TabLoRA lies between these two extremes: it shares the main backbone weights across predictors and introduces predictor\-specific low\-rank adaptations\.

The first part of this ablation compares the predictive behavior of Single, Full, and TabLoRA after hyperparameter optimization \(HPO\)\. As shown in Fig\.[7](https://arxiv.org/html/2607.10077#S5.F7), Full and TabLoRA achieve the same average rank of1\.5±0\.51\.5\\pm 0\.5, while Single obtains the worst rank on every dataset\. This indicates that ensemble\-style modeling clearly improves over the single\-predictor baseline\. The relative improvement plot further shows that both Full and TabLoRA consistently improve over Single\. Their improvement distributions are broadly comparable, indicating that TabLoRA can achieve full\-ensemble\-level predictive gains while avoiding full backbone duplication\.

![Refer to caption](https://arxiv.org/html/2607.10077v1/x7.png)Figure 7:Comparison of Single, Full, and TabLoRA\. The left panel shows average ranks, and the right panel shows relative improvement over Single\. Diamonds denote mean improvements\.The second part isolates the structural parameter efficiency of TabLoRA\. Since the HPO\-selected configurations may use different hidden dimensions and depths, their parameter counts do not provide a clean architecture\-level comparison\. Therefore, we conduct a fixed\-configuration parameter scaling analysis\. We fix the input dimension to 500 and vary the hidden dimension and the number of backbone blocks\. These correspond to the size of the hidden weightsWℓW^\{\\ell\}and the depthLLin Fig\.[1](https://arxiv.org/html/2607.10077#S3.F1)\. For each configuration, we compute the number of trainable parameters, measured in millions, for Full and TabLoRA, and report the parameter ratio of TabLoRA relative to Full\.

Fig\.[8](https://arxiv.org/html/2607.10077#S5.F8)shows that Full Ensemble parameters grow rapidly as the hidden dimension and depth increase\. In contrast, TabLoRA grows much more slowly because it avoids duplicating the full backbone for each predictor\. The right panel further confirms this trend: the parameter ratio of TabLoRA to Full decreases as the hidden dimension increases, especially for deeper networks\. This supports the complexity analysis in[3\.6](https://arxiv.org/html/2607.10077#S3.SS6)and shows that the low\-rank ensemble design provides structural parameter efficiency\.

![Refer to caption](https://arxiv.org/html/2607.10077v1/x8.png)Figure 8:Fixed\-configuration parameter scaling\. The left panel shows trainable parameters of Full and TabLoRA under different hidden dimensions and depths\. The right panel reports the parameter ratio of TabLoRA to Full\.

### 5\.4Performance on Small\- and Medium\-Scale Datasets

To further evaluate the behavior of TabLoRA beyond large\-scale settings, we conduct additional experiments on 50 small\- to medium\-scale datasets from CC18222[https://www\.openml\.org/search?type=benchmark&sort=tasks\_included&study\_type=task&id=99](https://www.openml.org/search?type=benchmark&sort=tasks_included&study_type=task&id=99)and CTR23333[https://www\.openml\.org/search?type=benchmark&sort=tasks\_included&study\_type=task&id=353](https://www.openml.org/search?type=benchmark&sort=tasks_included&study_type=task&id=353)\. These datasets satisfy the following constraints: the number of samples is no more than 50,000, the number of classes is no more than 10 for classification tasks, and the number of features is no more than 2,000\. Fig\.[9](https://arxiv.org/html/2607.10077#S5.F9)summarizes the average performance ranks and the relative improvement over MLP, while the complete numerical results are provided in Table[6](https://arxiv.org/html/2607.10077#A3.T6)\.

The results show that TabPFN performs particularly well in this setting, achieving the best average rank among all compared methods\. This is consistent with the design of TabPFN as a foundation\-model approach for tabular prediction, especially in small\-data regimes\[[19](https://arxiv.org/html/2607.10077#bib.bib54),[20](https://arxiv.org/html/2607.10077#bib.bib74)\]\. The three GBDT methods also remain strong and stable baselines, with XGBoost, CatBoost, and LightGBM ranking immediately after TabPFN\.

Compared with these strong baselines, TabLoRA does not achieve the best average rank\. However, the relative improvement plot shows that TabLoRA still provides clear gains over the plain MLP baseline on many datasets\. This indicates that the proposed low\-rank ensemble design is beneficial beyond large\-scale datasets, although its strongest advantage is not in outperforming TabPFN on small and medium\-scale data\. Instead, these results suggest that TabLoRA remains a competitive neural tabular model, while the main benefit of the method lies in scalable trainable ensemble learning and practical efficiency on large\-scale datasets\.

![Refer to caption](https://arxiv.org/html/2607.10077v1/x9.png)Figure 9:Performance on 50 small\- to medium\-scale datasets\. The left panel reports average ranks, where lower is better, and the right panel shows dataset\-level relative improvement over MLP\.

## 6Conclusions

In this paper, we proposed TabLoRA, a parameter\-efficient trainable neural ensemble framework for large\-scale tabular learning\. Motivated by the computational challenges of applying strong tabular models to large\-scale datasets, TabLoRA constructs an ensemble using a shared backbone and predictor\-specific low\-rank adaptations\. This design enables ensemble\-style prediction without duplicating the full backbone for each predictor\. Extensive experiments show that TabLoRA maintains a stable GPU memory footprint while remaining competitive with strong GBDT methods, recent neural tabular models, and a pretrained tabular reference model\. Ablation studies confirm the effectiveness of both the feature adapter and the low\-rank ensemble parameterization, showing that TabLoRA can preserve much of the benefit of full ensembles while reducing parameter growth\. Overall, these results suggest that parameter\-efficient ensemble design is a promising direction for practical tabular deep learning\.

## Declaration of generative AI use

During the preparation of this work the authors used ChatGPT in order to improve the language and readability\. After using this tool/service, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication\.

## Appendix ADatasets Description

Table\.[2](https://arxiv.org/html/2607.10077#A1.T2)lists the datasets used in this paper, the column \#Target means the number of distinct values in the label\.

Table 2:Details of datasets\. \#Target in regression task means the number of unique values\.
## Appendix BOptimization of hyperparameters

Table 3:Hyperparameters space\.HyperParametersRangeHyperParametersRangeXGBoostn​u​m​\_​b​o​o​s​t​\_​r​o​u​n​dnum\\\_boost\\\_round200e​a​r​l​y​\_​s​t​o​p​p​i​n​g​\_​r​o​u​n​d​searly\\\_stopping\\\_rounds20m​a​x​\_​d​e​p​t​hmax\\\_depthLogUniformInt \[2, 10\]a​l​p​h​aalphaLogUniform \[1e\-8, 0\.1\]l​a​m​b​d​alambdaLogUniform \[0\.5, 2\]e​t​aetaLogUniform \[0\.05, 0\.3\]CatBoosti​t​e​r​a​t​i​o​n​siterations200o​d​\_​w​a​i​tod\\\_wait20m​a​x​\_​d​e​p​t​hmax\\\_depthLogUniformInt \[2, 10\]l​2​\_​l​e​a​f​\_​r​e​gl2\\\_leaf\\\_regLogUniform \[0\.1, 2\]l​e​a​r​n​i​n​g​\_​r​a​t​elearning\\\_rateLogUniform \[0\.05, 0\.3\]LightGBMi​t​e​r​a​t​i​o​n​siterations200e​a​r​l​y​\_​s​t​o​p​p​i​n​g​\_​r​o​u​n​dearly\\\_stopping\\\_round20n​u​m​\_​l​e​a​v​e​snum\\\_leavesLogUniformInt \[8, 48\]l​a​m​b​d​a​\_​l1lambda\\\_l\_\{1\}LogUniform \[1e\-8, 0\.1\]l​a​m​b​d​a​\_​l2lambda\\\_l\_\{2\}LogUniform \[1e\-8, 0\.1\]l​e​a​r​n​i​n​g​\_​r​a​t​elearning\\\_rateLogUniform \[0\.05, 0\.3\]MLPn​\_​l​a​y​e​r​sn\\\_layersUniformInt \[1, 9\]d​\_​h​i​d​d​e​nd\\\_hiddenUniformInt \[64, 512\], step=32d​r​o​p​o​u​tdropout\[0, 0\.5\]RealMLPn​\_​b​l​o​c​k​sn\\\_blocksUniformInt \[1, 10\]h​i​d​d​e​n​\_​s​i​z​e​shidden\\\_sizes\[64, 256, 512\]d​r​o​p​o​u​tdropout\[0, 0\.15, 0\.3\]NCARTn​\_​t​r​e​e​sn\\\_trees\[8, 16, 32, 64\]n​\_​s​e​l​e​c​t​e​dn\\\_selectedUniformInt \[2, 10\]n​\_​l​a​y​e​r​sn\\\_layers\[2, 4\]m​a​s​k​\_​t​y​p​emask\\\_type\[sparsemax, entmax\]TabMn​\_​b​i​n​sn\\\_binsUniformInt \[2, 128\]d​\_​e​m​b​e​d​d​i​n​gd\\\_embeddingUniformInt \[8, 32\],step=4n​\_​b​l​o​c​k​sn\\\_blocksUniformInt \[1, 4\]d​\_​b​l​o​c​k​sd\\\_blocksUniformInt \[64, 1024\],step=16d​r​o​p​o​u​tdropout\[0, 0\.5\]kk32TabLoRAd​\_​h​i​d​d​e​nd\\\_hiddenUniformInt \[64, 512\], step=32d​r​o​p​o​u​tdropout\[0, 0\.5\]n​\_​b​l​o​c​k​sn\\\_blocksUniformInt \[1, 9\]r​\_​r​a​n​kr\\\_rankUniformInt \[4, 16\],step=2KK32

## Appendix CMore results

Table 4:Practical memory usage \(MB\)\. Theboldindicates the top result;OOMrepresents there exists GPU overflowTable 5:Mean results of the ablation studies\.Rawdenotes the TabLoRA variant without the feature adapter, andImprov\.denotes the improvement of TabLoRA over Raw\.Singledenotes a single\-predictor model, andFulldenotes the full deep ensemble with independent predictors\.Table 6:Mean results of 9 models on 50 datasets\. Theboldindicates the top result\.
## References

- \[1\]T\. Akiba, S\. Sano, T\. Yanase, T\. Ohta, and M\. Koyama\(2019\)Optuna: a next\-generation hyperparameter optimization framework\.InProceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining,pp\. 2623–2631\.Cited by:[§4\.3](https://arxiv.org/html/2607.10077#S4.SS3.p1.1)\.
- \[2\]S\. Ö\. Arik and T\. Pfister\(2021\)Tabnet: attentive interpretable tabular learning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.35,pp\. 6679–6687\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px2.p1.1)\.
- \[3\]V\. Borisov, T\. Leemann, K\. Seßler, J\. Haug, M\. Pawelczyk, and G\. Kasneci\(2022\)Deep neural networks and tabular data: a survey\.IEEE Transactions on Neural Networks and Learning Systems\.Cited by:[§1](https://arxiv.org/html/2607.10077#S1.p1.1)\.
- \[4\]J\. Chen, K\. Liao, Y\. Fang, D\. Chen, and J\. Wu\(2022\)TabCaps: a capsule neural network for tabular data classification with bow routing\.InThe Eleventh International Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px3.p1.1)\.
- \[5\]J\. Chen, K\. Liao, Y\. Wan, D\. Z\. Chen, and J\. Wu\(2022\)Danets: deep abstract networks for tabular data classification and regression\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.36,pp\. 3930–3938\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px3.p1.1)\.
- \[6\]J\. Chen, J\. Yan, Q\. Chen, D\. Z\. Chen, J\. Wu, and J\. Sun\(2024\)Can a deep learning model be a sure bet for tabular prediction?\.InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 288–296\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px2.p1.1)\.
- \[7\]T\. Chen and C\. Guestrin\(2016\)Xgboost: a scalable tree boosting system\.InProceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining,pp\. 785–794\.Cited by:[§1](https://arxiv.org/html/2607.10077#S1.p1.1),[§4\.1](https://arxiv.org/html/2607.10077#S4.SS1.p1.1)\.
- \[8\]T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer\(2023\)Qlora: efficient finetuning of quantized llms\.Advances in neural information processing systems36,pp\. 10088–10115\.Cited by:[§2\.2](https://arxiv.org/html/2607.10077#S2.SS2.p2.1)\.
- \[9\]N\. Ding, Y\. Qin, G\. Yang, F\. Wei, Z\. Yang, Y\. Su, S\. Hu, Y\. Chen, C\. Chan, W\. Chen,et al\.\(2023\)Parameter\-efficient fine\-tuning of large\-scale pre\-trained language models\.Nature machine intelligence5\(3\),pp\. 220–235\.Cited by:[§2\.2](https://arxiv.org/html/2607.10077#S2.SS2.p1.1)\.
- \[10\]Y\. Gorishniy, A\. Kotelnikov, and A\. Babenko\(2024\)Tabm: advancing tabular deep learning with parameter\-efficient ensembling\.arXiv preprint arXiv:2410\.24210\.Cited by:[§1](https://arxiv.org/html/2607.10077#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2607.10077#S4.SS1.p1.1)\.
- \[11\]Y\. Gorishniy, I\. Rubachev, and A\. Babenko\(2022\)On embeddings for numerical features in tabular deep learning\.Advances in Neural Information Processing Systems35,pp\. 24991–25004\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px1.p1.1)\.
- \[12\]Y\. Gorishniy, I\. Rubachev, N\. Kartashev, D\. Shlenskii, A\. Kotelnikov, and A\. Babenko\(2024\)Tabr: tabular deep learning meets nearest neighbors\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 18209–18249\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px1.p1.1)\.
- \[13\]Y\. Gorishniy, I\. Rubachev, V\. Khrulkov, and A\. Babenko\(2021\)Revisiting deep learning models for tabular data\.Advances in Neural Information Processing Systems34,pp\. 18932–18943\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px2.p1.1)\.
- \[14\]L\. Grinsztajn, K\. Flöge, O\. Key, F\. Birkel, P\. Jund, B\. Roof, B\. Jäger, D\. Safaric, S\. Alessi, A\. Hayler,et al\.\(2025\)Tabpfn\-2\.5: advancing the state of the art in tabular foundation models\.arXiv preprint arXiv:2511\.08667\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px4.p1.1)\.
- \[15\]L\. Grinsztajn, K\. Flöge, O\. Key, F\. Birkel, P\. Jund, B\. Roof, M\. Manium, S\. Bin, M\. Bühler, A\. Garg,et al\.\(2026\)TabPFN\-3: technical report\.arXiv preprint arXiv:2605\.13986\.Cited by:[§1](https://arxiv.org/html/2607.10077#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px4.p1.1),[§4\.1](https://arxiv.org/html/2607.10077#S4.SS1.p1.1)\.
- \[16\]L\. Grinsztajn, E\. Oyallon, and G\. Varoquaux\(2022\)Why do tree\-based models still outperform deep learning on typical tabular data?\.InThirty\-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track,Cited by:[§1](https://arxiv.org/html/2607.10077#S1.p1.1)\.
- \[17\]E\. Hajiramezanali, N\. L\. Diamant, G\. Scalia, and M\. W\. Shen\(2022\)Stab: self\-supervised learning for tabular data\.InNeurIPS 2022 First Table Representation Workshop,Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px3.p1.1)\.
- \[18\]S\. Hegselmann, A\. Buendia, H\. Lang, M\. Agrawal, X\. Jiang, and D\. Sontag\(2023\)Tabllm: few\-shot classification of tabular data with large language models\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 5549–5581\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px3.p1.1)\.
- \[19\]N\. Hollmann, S\. Müller, K\. Eggensperger, and F\. Hutter\(2022\)Tabpfn: a transformer that solves small tabular classification problems in a second\.arXiv preprint arXiv:2207\.01848\.Cited by:[§1](https://arxiv.org/html/2607.10077#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px4.p1.1),[§5\.4](https://arxiv.org/html/2607.10077#S5.SS4.p2.1)\.
- \[20\]N\. Hollmann, S\. Müller, L\. Purucker, A\. Krishnakumar, M\. Körfer, S\. B\. Hoo, R\. T\. Schirrmeister, and F\. Hutter\(2025\)Accurate predictions on small data with a tabular foundation model\.Nature637\(8045\),pp\. 319–326\.Cited by:[§1](https://arxiv.org/html/2607.10077#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px4.p1.1),[§5\.4](https://arxiv.org/html/2607.10077#S5.SS4.p2.1)\.
- \[21\]D\. Holzmüller, L\. Grinsztajn, and I\. Steinwart\(2024\)Better by default: strong pre\-tuned mlps and boosted trees on tabular data\.Advances in Neural Information Processing Systems37,pp\. 26577–26658\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2607.10077#S4.SS1.p1.1)\.
- \[22\]N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. De Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. Gelly\(2019\)Parameter\-efficient transfer learning for nlp\.InInternational conference on machine learning,pp\. 2790–2799\.Cited by:[§2\.2](https://arxiv.org/html/2607.10077#S2.SS2.p2.1),[§3\.2](https://arxiv.org/html/2607.10077#S3.SS2.p1.3)\.
- \[23\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.Iclr1\(2\),pp\. 3\.Cited by:[§2\.2](https://arxiv.org/html/2607.10077#S2.SS2.p2.1),[§3\.3](https://arxiv.org/html/2607.10077#S3.SS3.p1.3)\.
- \[24\]X\. Huang, A\. Khetan, M\. Cvitkovic, and Z\. Karnin\(2020\)Tabtransformer: tabular data modeling using contextual embeddings\.arXiv preprint arXiv:2012\.06678\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px2.p1.1)\.
- \[25\]J\. Jiang, S\. Liu, H\. Cai, Q\. Zhou, and H\. Ye\(2026\)Representation learning for tabular data: a comprehensive survey\.IEEE Transactions on Pattern Analysis and Machine Intelligence\.Cited by:[§1](https://arxiv.org/html/2607.10077#S1.p1.1)\.
- \[26\]A\. Kadra, M\. Lindauer, F\. Hutter, and J\. Grabocka\(2021\)Well\-tuned simple nets excel on tabular datasets\.Advances in neural information processing systems34,pp\. 23928–23941\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px1.p1.1)\.
- \[27\]G\. Ke, Q\. Meng, T\. Finley, T\. Wang, W\. Chen, W\. Ma, Q\. Ye, and T\. Liu\(2017\)Lightgbm: a highly efficient gradient boosting decision tree\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2607.10077#S1.p1.1),[§4\.1](https://arxiv.org/html/2607.10077#S4.SS1.p1.1)\.
- \[28\]J\. Kossen, N\. Band, C\. Lyle, A\. N\. Gomez, T\. Rainforth, and Y\. Gal\(2021\)Self\-attention between datapoints: going beyond individual input\-output pairs in deep learning\.Advances in Neural Information Processing Systems34,pp\. 28742–28756\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px2.p1.1)\.
- \[29\]R\. Levin, V\. Cherepanova, A\. Schwarzschild, A\. Bansal, C\. B\. Bruss, T\. Goldstein, A\. G\. Wilson, and M\. Goldblum\(2022\)Transfer learning with deep tabular models\.arXiv preprint arXiv:2206\.15306\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px3.p1.1)\.
- \[30\]J\. Luo and S\. Xu\(2024\)NCART: neural classification and regression tree for tabular data\.Pattern Recognition154,pp\. 110578\.Cited by:[§1](https://arxiv.org/html/2607.10077#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2607.10077#S4.SS1.p1.1)\.
- \[31\]J\. Ma, V\. Thomas, R\. Hosseinzadeh, A\. Labach, J\. Cresswell, K\. Golestan, G\. Yu, A\. L\. Caterini, and M\. Volkovs\(2026\)TabDPT: scaling tabular foundation models on real data\.Advances in Neural Information Processing Systems38,pp\. 172692–172722\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px4.p1.1)\.
- \[32\]S\. Popov, S\. Morozov, and A\. Babenko\(2019\)Neural oblivious decision ensembles for deep learning on tabular data\.arXiv preprint arXiv:1909\.06312\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px3.p1.1)\.
- \[33\]L\. Prokhorenkova, G\. Gusev, A\. Vorobev, A\. V\. Dorogush, and A\. Gulin\(2018\)CatBoost: unbiased boosting with categorical features\.Advances in neural information processing systems31\.Cited by:[§1](https://arxiv.org/html/2607.10077#S1.p1.1),[§4\.1](https://arxiv.org/html/2607.10077#S4.SS1.p1.1)\.
- \[34\]J\. Qu, D\. Holzmüller, G\. Varoquaux, and M\. L\. Morvan\(2025\)Tabicl: a tabular foundation model for in\-context learning on large data\.arXiv preprint arXiv:2502\.05564\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px4.p1.1)\.
- \[35\]J\. Qu, D\. Holzmüller, G\. Varoquaux, and M\. L\. Morvan\(2026\)TabICLv2: a better, faster, scalable, and open tabular foundation model\.arXiv preprint arXiv:2602\.11139\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px4.p1.1)\.
- \[36\]I\. Shavitt and E\. Segal\(2018\)Regularization learning networks: deep learning for tabular datasets\.Advances in Neural Information Processing Systems31\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px1.p1.1)\.
- \[37\]R\. Shwartz\-Ziv and A\. Armon\(2022\)Tabular data: deep learning is not all you need\.Information Fusion81,pp\. 84–90\.Cited by:[§1](https://arxiv.org/html/2607.10077#S1.p1.1)\.
- \[38\]G\. Somepalli, M\. Goldblum, A\. Schwarzschild, C\. B\. Bruss, and T\. Goldstein\(2021\)Saint: improved neural networks for tabular data via row attention and contrastive pre\-training\.arXiv preprint arXiv:2106\.01342\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px2.p1.1)\.
- \[39\]B\. Sun, L\. Yang, W\. Zhang, M\. Lin, P\. Dong, C\. Young, and J\. Dong\(2019\)Supertml: two\-dimensional word embedding for the precognition on structured tabular data\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,pp\. 0–0\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px3.p1.1)\.
- \[40\]T\. Ucar, E\. Hajiramezanali, and L\. Edwards\(2021\)Subtab: subsetting features of tabular data for self\-supervised representation learning\.Advances in Neural Information Processing Systems34,pp\. 18853–18865\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px3.p1.1)\.
- \[41\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px2.p1.1)\.
- \[42\]Y\. Wang, X\. Zhang, H\. Yu, M\. Ming, G\. Ren, H\. Yuan, L\. Mao, Y\. Zhang, C\. Yuan, and P\. Cui\(2026\)LimiX\-2m: mitigating low\-rank collapse and attention bottlenecks in tabular foundation models\.arXiv preprint arXiv:2606\.04485\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px4.p1.1)\.
- \[43\]Z\. Wang and J\. Sun\(2022\)Transtab: learning transferable tabular transformers across tables\.Advances in Neural Information Processing Systems35,pp\. 2902–2915\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px3.p1.1)\.
- \[44\]J\. Yan, J\. Chen, Y\. Wu, D\. Z\. Chen, and J\. Wu\(2023\)T2g\-former: organizing tabular features into relation graphs promotes heterogeneous feature interaction\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.37,pp\. 10720–10728\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px2.p1.1)\.
- \[45\]H\. Ye, H\. Yin, and D\. Zhan\(2024\)Modern neighborhood components analysis: a deep tabular baseline two decades later\.arXiv preprint arXiv:2407\.03257\.Cited by:[§1](https://arxiv.org/html/2607.10077#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px1.p1.1)\.
- \[46\]P\. Yin, G\. Neubig, W\. Yih, and S\. Riedel\(2020\)TaBERT: pretraining for joint understanding of textual and tabular data\.arXiv preprint arXiv:2005\.08314\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px3.p1.1)\.
- \[47\]J\. Yoon, Y\. Zhang, J\. Jordon, and M\. van der Schaar\(2020\)Vime: extending the success of self\-and semi\-supervised learning to tabular domain\.Advances in Neural Information Processing Systems33,pp\. 11033–11043\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px3.p1.1)\.
- \[48\]Q\. Zhang, M\. Chen, A\. Bukharin, N\. Karampatziakis, P\. He, Y\. Cheng, W\. Chen, and T\. Zhao\(2023\)Adalora: adaptive budget allocation for parameter\-efficient fine\-tuning\.arXiv preprint arXiv:2303\.10512\.Cited by:[§2\.2](https://arxiv.org/html/2607.10077#S2.SS2.p2.1)\.
- \[49\]X\. Zhang, G\. Ren, H\. Yu, H\. Yuan, H\. Wang, J\. Li, J\. Wu, L\. Mo, L\. Mao, M\. Hao,et al\.\(2025\)Limix: unleashing structured\-data modeling capability for generalist intelligence\.arXiv preprint arXiv:2509\.03505\.Cited by:[§2\.1](https://arxiv.org/html/2607.10077#S2.SS1.SSS0.Px4.p1.1)\.