Agentic Search Spaces for Tabular Machine Learning
Summary
The paper investigates using LLM-based agents to design expanded hyperparameter search spaces for tabular machine learning models, demonstrating performance improvements without extra tuning costs across multiple datasets.
View Cached Full Text
Cached at: 09/16/26, 08:47 AM
# Agentic Search Spaces for Tabular Machine Learning
Source: [https://arxiv.org/html/2609.16309](https://arxiv.org/html/2609.16309)
Renat Sergazinov†††thanks:Corresponding author, email at <mc\.sergazinov@gmail\.com\>†Yandex‡HSE UniversitySergey Pankevich†‡Artem Babenko†‡
###### Abstract
Despite the rapid progress of LLM\-based agents for planning, code generation, and debugging, their practical value for tabular machine learning remains underexplored\. In this paper, we investigate a concrete use case: whether state\-of\-the\-art agentic AI systems can design extended HPO search spaces for established tabular models that outperform the standard search spaces provided by the model authors\.
Specifically, we represent each tabular model as a modular pipeline covering preprocessing, embeddings, architecture, training, and inference\. We then task the agent to propose candidate code implementations for each module and use a classical HPO algorithm to jointly optimize over these candidates and the model’s default hyperparameters\. Compared with the base HPO spaces, the expanded search spaces improve the performance of nearly every model family across a suite of 45 datasets, with average relative gains of0\.6%0\.6\\%, rising to2\.0%2\.0\\%on small\-to\-medium regression datasets\. Notably, these gains come at no extra tuning cost: the enlarged spaces outperform the base under the same tuning and ensembling budgets\. The gains transfer to the recent TabArena benchmark, where the agentic spaces improve the official Elo scores of four of the five model families and the two strongest agentic ensembles surpass the best AutoGluon ensemble of conventional models\. Overall, our study suggests that LLM agents can provide practical value for tabular ML by expanding the design space\. The code is available at:[https://github\.com/yandex\-research/agentic\-hpset](https://github.com/yandex-research/agentic-hpset)\.
## 1Introduction
Tabular data is one of the most abundant modalities in machine learning, with applications across finance, healthcare, recommender systems, and scientific data analysis\. This practical importance has made tabular ML an active research area, with recent progress in specialized DNN architectures\[[1](https://arxiv.org/html/2609.16309#bib.bib15),[2](https://arxiv.org/html/2609.16309#bib.bib3),[3](https://arxiv.org/html/2609.16309#bib.bib4)\], foundational models\[[4](https://arxiv.org/html/2609.16309#bib.bib6),[5](https://arxiv.org/html/2609.16309#bib.bib5),[6](https://arxiv.org/html/2609.16309#bib.bib7)\], and more rigorous benchmarks and evaluation protocols\[[7](https://arxiv.org/html/2609.16309#bib.bib12),[8](https://arxiv.org/html/2609.16309#bib.bib11)\]\.
At the same time, the recent rise of large language models \(LLMs\) has so far had only limited impact on state\-of\-the\-art tabular ML\. While prior work has investigated LLMs for direct prediction over serialized tables\[[9](https://arxiv.org/html/2609.16309#bib.bib26),[10](https://arxiv.org/html/2609.16309#bib.bib29)\], semantic feature engineering\[[11](https://arxiv.org/html/2609.16309#bib.bib27),[12](https://arxiv.org/html/2609.16309#bib.bib28)\], and prediction with textual metadata\[[13](https://arxiv.org/html/2609.16309#bib.bib37),[14](https://arxiv.org/html/2609.16309#bib.bib36)\], these approaches appear most useful in specific niches, e\.g\., when datasets are small or feature names are semantically informative\. For more general cases, however, LLMs have not substantially affected standard practitioner workflows in tabular ML\.
Recent developments in agentic AI systems suggest new possible roles for LLMs in tabular problems\. In particular, the ability of advanced AI agents to analyse and improve code is well suited for an under\-automated part of tabular ML practice:*identifying which recent techniques or combinations of them can improve a particular model*\. Most tabular models are released as a frozen recipe with a small hyperparameter \(HPO\) set\. In practice, however, users frequently hand\-tune models based on domain knowledge and current literature trends \(e\.g\. new activation functions or an optimizer\) with no systematic way to explore different options or their interactions\. In this paper, we investigate whether state\-of\-the\-art agents can close this gap automatically\. Given a model, we split it into a sequence of code modules \(e\.g\. preprocessing, embedding, architecture\) and task the agent to propose and implement in code the alternatives for each of them\. These code modules are then added as categorical variables to the HPO space\. We visualize our proposed method in Figure[1](https://arxiv.org/html/2609.16309#S1.F1)\. The model is then tuned over this extended search space using a standard hyperparameter optimization \(HPO\)\. This use of LLMs is attractive for several reasons\. First, it improves performance: we find the enlarged search space lets HPO find model variants that outperform the model tuned over the default author\-provided HPO space\. Second, it simplifies model upkeep: practitioners no longer need to find and test each new idea manually\. Third, the cost of the LLM can be amortized across the datasets: the search space can be generated once per\-model without access to the target data\.
We evaluate agent\-augmented HPO search spaces across representative tabular model families, including DNNs, GBDTs, and tabular foundation models, on a curated set of 45 datasets\. Compared to the default HPO spaces, the agent\-augmented spaces improve the aggregate rank of nearly every family\. Averaged across all datasets, tuned single models gain0\.5%0\.5\\%for DNNs \(MLP†, TabM†, RealMLP\),0\.8%0\.8\\%for GBDTs \(LightGBM\), and0\.5%0\.5\\%for the TabICLv2 foundation model, while ensembles gain0\.9%0\.9\\%for DNNs and0\.2%0\.2\\%for GBDTs\. The gains concentrate on small\-to\-medium regression datasets, where the average single\-model gains rise to2\.0%2\.0\\%for DNN and GBDT, while TabICLv2 improves on large regression datasets by2\.1%2\.1\\%\. The richer spaces also improve hyperparameter ensembling, reducing prediction correlation and yielding stronger ensembles across the board\. Finally, the conclusions transfer to the recent TabArena benchmark, where the agentic spaces improve the Elo scores of four of the five model families, by up to125125points\.
Overall, our results suggest a practical and complementary role for LLM agents in tabular ML\. Rather than replacing specialized tabular models or performing expensive dataset\-specific experimentation, agents can act as automated search\-space designers that transfer broad ML implementation knowledge into reusable, model\-specific HPO spaces\.
Base modelMLP, TabM, GBDTLLM agentGenerates per moduleCandidate setsℋ1\\mathcal\{H\}\_\{1\}preprocessingℋ2\\mathcal\{H\}\_\{2\}embeddingℋ3\\mathcal\{H\}\_\{3\}architectureℋ4\\mathcal\{H\}\_\{4\}trainingℋ5\\mathcal\{H\}\_\{5\}inferenceℋ1×⋯×ℋ5×𝒮HPO\\mathcal\{H\}\_\{1\}\\\!\\times\\\!\\cdots\\\!\\times\\\!\\mathcal\{H\}\_\{5\}\\\!\\times\\\!\\mathcal\{S\}\_\{\\mathrm\{HPO\}\}Joint search spaceBayesian HPOSelects joint configNew datasetTrain/val splitsFitted pipelineBest joint configStage 1 — per model \(run once\)Stage 2 — per dataset \(run each time\)
Figure 1:Method overview\.*Stage 1*runs once per base model: an LLM agent generates candidate setsℋ1,…,ℋ5\\mathcal\{H\}\_\{1\},\\dots,\\mathcal\{H\}\_\{5\}for the five pipeline modules\.*Stage 2*runs on each new dataset: a classical HPO searches the joint space and returns a fitted pipeline\.The contributions of our paper are the following:
1. 1\.We propose a new method for improving tabular models with LLM agents: the researcher splits the model into a sequence of modules, the agent generates candidate module implementations, and a traditional HPO algorithm explores the resulting joint search space on each dataset\. We make our prompts and harnesses publicly available\.
2. 2\.We test the method by generating extended HPO search spaces, which we also release, for five model families: MLP†, TabM†, LightGBM, RealMLP, and TabICLv2\.
3. 3\.On these models, we show consistent average gains across the 45 datasets:0\.5%0\.5\\%for DNNs,0\.8%0\.8\\%for GBDTs, and0\.5%0\.5\\%for TabICLv2 in the single\-model regime, and0\.9%0\.9\\%for DNNs in the ensemble regime with up to2\.0%2\.0\\%on small\-to\-medium regression tasks\. The ensemble over the agentic RealMLP space is the strongest of all model families in our study, outperformed only by the multi\-model AutoGluon system\. We also evaluate on the TabArena benchmark and observe consistent gains there as well\.
## 2Related Works
Tabular deep learning and pipeline design\.Recent tabular work has advanced both architectures \(TabM\[[2](https://arxiv.org/html/2609.16309#bib.bib3)\], TabR\[[15](https://arxiv.org/html/2609.16309#bib.bib8)\], ModernNCA\[[3](https://arxiv.org/html/2609.16309#bib.bib4)\]\) and foundation models \(TabPFN\[[4](https://arxiv.org/html/2609.16309#bib.bib6)\], TabICLv2\[[16](https://arxiv.org/html/2609.16309#bib.bib38)\], LimiX\[[6](https://arxiv.org/html/2609.16309#bib.bib7)\]\), while benchmarks like TabReD\[[7](https://arxiv.org/html/2609.16309#bib.bib12)\]and TabArena\[[8](https://arxiv.org/html/2609.16309#bib.bib11)\]show that protocol choices substantially affect conclusions\.
AutoML and architecture search\.Auto\-sklearn\[[17](https://arxiv.org/html/2609.16309#bib.bib14)\]builds a greedy ensemble\[[18](https://arxiv.org/html/2609.16309#bib.bib70)\]over combinations of preprocessing methods and shallow models\. AutoGluon\[[19](https://arxiv.org/html/2609.16309#bib.bib13)\]follows the same ensemble\-first recipe, but does not optimize over preprocessing and broadens the model library from shallow learners to deep architectures \(e\.g\. TabM\[[2](https://arxiv.org/html/2609.16309#bib.bib3)\], RealMLP\[[1](https://arxiv.org/html/2609.16309#bib.bib15)\]\) and foundation models \(e\.g\. TabICLv2\[[16](https://arxiv.org/html/2609.16309#bib.bib38)\], LimiX\[[6](https://arxiv.org/html/2609.16309#bib.bib7)\]\)\. AutoPyTorch\[[20](https://arxiv.org/html/2609.16309#bib.bib19)\]instead ensembles parametrizations of a single model family, varying regularizers, activations, and MLP layer shapes\. AutoKeras\[[21](https://arxiv.org/html/2609.16309#bib.bib20)\]similarly searches over neural architectures assembled from its own library of implemented components\[[22](https://arxiv.org/html/2609.16309#bib.bib17),[23](https://arxiv.org/html/2609.16309#bib.bib18)\]\. Our setting is closest to AutoPyTorch and AutoKeras: we start from a single model and search over its modifications\. We differ in two key ways\. First, the candidate generation is conditioned on the input model: the agent proposes modules specifically for the given model\. Second, the candidates are written by the agent rather than hand\-implemented\. The resulting models with extended HPO spaces can then be wrapped and used with Auto\-sklearn and AutoGluon\.
LLMs for tabular prediction\.Prior LLM\-based approaches to tabular prediction fall into two families\. The first treats the LLM as a direct predictor over serialized rows, via prompting \(TabLLM\[[9](https://arxiv.org/html/2609.16309#bib.bib26)\]\) or fine\-tuning \(Tabula\-8B\[[10](https://arxiv.org/html/2609.16309#bib.bib29)\]\); the second uses the LLM for semantic feature engineering, generating new columns from dataset descriptions \(CAAFE\[[11](https://arxiv.org/html/2609.16309#bib.bib27)\], FeatLLM\[[12](https://arxiv.org/html/2609.16309#bib.bib28)\]\)\. Both have shown gains in regimes such as small datasets or rich textual metadata\.
LLM agents for optimization and research automation\.Prior work uses LLMs either as optimizers\[[24](https://arxiv.org/html/2609.16309#bib.bib21),[25](https://arxiv.org/html/2609.16309#bib.bib22),[26](https://arxiv.org/html/2609.16309#bib.bib23),[27](https://arxiv.org/html/2609.16309#bib.bib24),[28](https://arxiv.org/html/2609.16309#bib.bib25)\]or as agents that automate parts of the research loop\[[29](https://arxiv.org/html/2609.16309#bib.bib31),[30](https://arxiv.org/html/2609.16309#bib.bib2),[31](https://arxiv.org/html/2609.16309#bib.bib1)\]\. The former is orthogonal to our work: these methods change how a fixed search space is explored, whereas we use a standard sampler and change the space itself\. Our work is closer to the latter, but restricts the agent to generating modules along predefined pipeline axes, followed by controlled HPO\. This preserves dataset privacy, amortizes agent cost, and makes the resulting search spaces reproducible and amenable to ablation\.
## 3Methodology
### 3\.1Setup and notation
Datasets\.We consider a tabular task with a fixed split𝒟=\(𝒟train,𝒟val,𝒟test\)\\mathcal\{D\}=\(\\mathcal\{D\}\_\{\\mathrm\{train\}\},\\mathcal\{D\}\_\{\\mathrm\{val\}\},\\mathcal\{D\}\_\{\\mathrm\{test\}\}\)\. We use datasets derived from TabM\[[2](https://arxiv.org/html/2609.16309#bib.bib3)\], TabArena\[[8](https://arxiv.org/html/2609.16309#bib.bib11)\], and TabReD\[[7](https://arxiv.org/html/2609.16309#bib.bib12)\]benchmarks, which span both regression and classification datasets ranging from 768 to 1M\+ objects and from 5 to 1500\+ features\. We report their summary statistics in Table[1](https://arxiv.org/html/2609.16309#S3.T1)and list their full properties in Appendix[B\.2](https://arxiv.org/html/2609.16309#A2.SS2)\. Our design choices for the benchmark were to: \(1\) include both i\.i\.d\. and non\-i\.i\.d\. datasets; \(2\) keep a roughly balanced split between regression and classification; \(3\) include datasets with a diverse range of sample\-to\-feature ratios\. In addition, we separately report our model performance on the TabArena benchmark in Appendix[A](https://arxiv.org/html/2609.16309#A1)\.
Table 1:Overview of the datasets included in the main evaluation; sizes are total row counts\. Full per\-dataset statistics are in Appendix[B](https://arxiv.org/html/2609.16309#A2)\.Source\#Rows\#FeaturesTask typeOpenMLTabReDMin\.Q50MeanMax\.Min\.Q50MeanMax\.\#Regr\.\#Classif\.37876821K103K1\.2M52115617762025Metrics\.We evaluate every dataset with its native metric adopted from the source benchmark \(AUROC, accuracy, or log\-loss for classification and RMSE for regression\); the per\-dataset metrics are listed in Appendix[B\.2](https://arxiv.org/html/2609.16309#A2.SS2)\. We report four aggregate metrics, all at a fixed tuning budgetBB: mean rank, relative improvement over the tuned base\-space MLP†, normalized score, and Elo score\. We define these metrics in Appendix[B\.3](https://arxiv.org/html/2609.16309#A2.SS3)\.
Models and agents\.We select strong within\-family implementations\. To represent DNNs, we take, in order of increasing complexity: \(i\) MLP†– an MLP with piecewise\-linear embeddings of[Gorishniy et al\. \[32\]](https://arxiv.org/html/2609.16309#bib.bib9); \(ii\) TabM†– the mini variant of TabM with piecewise\-linear embeddings andk=32k=32members from[Gorishniy et al\. \[2\]](https://arxiv.org/html/2609.16309#bib.bib3); \(iii\) RealMLP – the MLP model from PyTabKit with the latest TabArena hyperparameters\[[1](https://arxiv.org/html/2609.16309#bib.bib15),[8](https://arxiv.org/html/2609.16309#bib.bib11)\]\. To represent GBDT, we take the LightGBM model\[[33](https://arxiv.org/html/2609.16309#bib.bib33)\]\. Among foundational models, we take the recently proposed TabICLv2\[[16](https://arxiv.org/html/2609.16309#bib.bib38)\]; since it performs no gradient\-based training, we refit it on the concatenated train and validation splits to obtain test predictions \(Appendix[B\.4](https://arxiv.org/html/2609.16309#A2.SS4)\)\. As an agent, we use Claude Code running Opus 4\.8 onmaxeffort settings\[[34](https://arxiv.org/html/2609.16309#bib.bib71),[35](https://arxiv.org/html/2609.16309#bib.bib75)\]\. We additionally run speed and performance ablations with Codex running GPT 5\.5 withextra\-highsettings\[[36](https://arxiv.org/html/2609.16309#bib.bib77),[37](https://arxiv.org/html/2609.16309#bib.bib76)\]\.
Evaluation setup\.We tune each model over its HPO space with Optuna’s univariate Tree\-structured Parzen Estimator \(TPE\) sampler\[[38](https://arxiv.org/html/2609.16309#bib.bib16),[39](https://arxiv.org/html/2609.16309#bib.bib32)\]\. The first2020trials are sampled at random to seed the densities\. Each trial trains one configuration on the training subset and scores it on the validation subset; the budget is200200trials on small and medium datasets and100100on the large ones \(Microsoft and TabReD\), after which the configuration with the best validation score is evaluated on the held\-out test subset with 15 random seeds\. TabICLv2 is the exception: the default model is used as published, without tuning, while its agentic variant is tuned with a fixed budget of100100trials on every dataset\. For the ensembling study, we build greedy ensembles following[Caruana et al\. \[18\]](https://arxiv.org/html/2609.16309#bib.bib70): we sample100100random configurations per search space \(2020on large datasets\), train them, and iteratively add, with replacement, the model whose inclusion in the prediction average most reduces the validation error\. The selected ensemble is then evaluated on the test subset; further details on the greedy ensemble algorithm are given in Appendix[B\.5](https://arxiv.org/html/2609.16309#A2.SS5)\.
### 3\.2From end\-to\-end agents to agent\-generated search spaces
Table 2:Selected agent\-proposed candidate implementations per module\. Tags\(reg\.\),\(clf\.\),\(cat\.\), and\(num\.\)mark candidates applicable only to regression, classification, categorical features, or numerical features, respectively\. The trailing number in parentheses is the count of agent\-generated candidates for that module; full per\-family lists are in Appendix[D](https://arxiv.org/html/2609.16309#A4)\.The most direct way to hand a tabular model to an agent is to let it edit the model end to end: starting from the reference implementation, the agent changes a hyperparameter or a piece of code, runs a validation fit, and keeps the change if the score improves\. This is known as the autoresearch loop described in[Karpathy \[30\]](https://arxiv.org/html/2609.16309#bib.bib2), which we adapt to the tabular setting and run for5050iterations on four datasets, starting from theMLP†\\mathrm\{MLP\}^\{\\dagger\}code \(Appendix[D\.3](https://arxiv.org/html/2609.16309#A4.SS3)\)\. Table[3](https://arxiv.org/html/2609.16309#S3.T3)summarizes the outcome: the loop costs $20–50*per dataset*, requires handing the agent the data, and ends up marginally behind classical HPO over the default HPO space\.
Table 3:The end\-to\-end autoresearch loop\[[30](https://arxiv.org/html/2609.16309#bib.bib2)\]against classical HPO and our method, on the four datasets of Table[16](https://arxiv.org/html/2609.16309#A4.T16)at a matched budget of5050trials\.Δ\\Deltais the mean relative improvement\.Three problems\.We attribute this negative result to three properties of the end\-to\-end loop\.
*\(i\) Few ideas generalize well in tabular data; diversity and number are therefore an advantage\.*The winning module combinations are pairwise distinct across the 18 datasets we examine \(Section[5\.1](https://arxiv.org/html/2609.16309#S5.SS1)\), and expanding the candidate pool monotonically grows performance \(Section[5\.3](https://arxiv.org/html/2609.16309#S5.SS3)\)\. No single module generalizes well across the datasets, so the search should maximize the*diversity*and*number*of explored ideas, covering as much of the idea space as possible\. Contrary to this, the end\-to\-end loop tends to converge on a narrow set of techniques\. Indeed LLMs are known to explore poorly in\-context\[[41](https://arxiv.org/html/2609.16309#bib.bib79)\], and agents in long loops increasingly revisit earlier actions as their context fills up\[[42](https://arxiv.org/html/2609.16309#bib.bib80),[43](https://arxiv.org/html/2609.16309#bib.bib81)\]\.
*\(ii\) The search space is explored inefficiently\.*Even for the ideas it does try, the greedy LLM\-based search is a poor search strategy: it evaluates one edit at a time, accepts greedily, does not maintain a model of the response surface, and proposes candidates based purely on the LLM prior\. The weakness is sharpest for the numerical hyperparameters, which hold a large share of the achievable gain \(learning rate, weight decay; Figure[3](https://arxiv.org/html/2609.16309#S5.F3), blue boxes\)\. On exactly such problems, classical black\-box optimization algorithms such as TPE sampler \(implemented in Optuna\) outperform LLMs\[[31](https://arxiv.org/html/2609.16309#bib.bib1)\]\.
*\(iii\) The cost does not amortize across datasets\.*The loop is re\-run for every new dataset, so its price scales with the benchmark\. Repeating our 45\-dataset study this way would cost at least $900–$2,300 in agent calls alone\. The agent also needs the data to score its own edits, which invites contamination\.
Three fixes\.Our method addresses each problem in turn\.
*\(i\) Steer the agent towards coverage\.*We decompose each model by hand into a diverse set of mutually non\-overlapping modules supported by the literature\. For the neural families \(MLP†, TabM†, and RealMLP\), we take the modules to be preprocessing, feature embeddings, architecture, training, and inference with the default taken from the original method, e\.g\., for MLP†quantile\-transform as numerical preprocessing, AdamW\[[44](https://arxiv.org/html/2609.16309#bib.bib54)\]as optimizer\. For LightGBM and TabICLv2, whose learning algorithms are frozen, we take the modules to be preprocessing and inference\. The agent proposes alternative implementations of each module as self\-contained code blocks, e\.g\., Yeo–Johnson feature transform\[[45](https://arxiv.org/html/2609.16309#bib.bib39)\], Lion optimizer\[[46](https://arxiv.org/html/2609.16309#bib.bib30)\]\. We instruct the agent to generate at most6464candidates per task type \(regression and classification\), asking it to filter out proposals that are repetitive, do not pass the evidence bar \(i\.e\., have been tried and failed in the literature before\), or repeatedly fail the synthetic smoke tests \(Appendix[D\.2](https://arxiv.org/html/2609.16309#A4.SS2)\)\. Each module then enters the search space as a categorical variable with a uniform prior, e\.g\.,train module∼Categorical\(\{AdamW,AdamW \+ EMA,…\}\)\\text\{train module\}\\sim\\mathrm\{Categorical\}\(\\\{\\text\{AdamW\},\\,\\text\{AdamW \+ EMA\},\\,\\dots\\\}\)\. Our design promotes high coverage of potentially promising model configurations\.
*\(ii\) Delegate the exploration to a classical sampler\.*The agent only generates module alternatives, expressed as code\. The joint space of modules and continuous hyperparameters is then explored by a standard HPO sampler\. Hence, the work is divided between the agent and the HPO algorithm\.
*\(iii\) Generate once per model family\.*The agent generates once per model family\. Concretely, it receives the model’s code, the compressed title and abstract of the original paper, and tools to test candidate implementations, but no target dataset\. The generation cost then amortizes over every dataset the space is later used on, and contamination or dataset leakage are excluded by construction\.
Table[2](https://arxiv.org/html/2609.16309#S3.T2)shows a selection of the proposed modules for MLP†; the full sets for each model and the complete agent environment are described in Appendix[D\.2](https://arxiv.org/html/2609.16309#A4.SS2)\.
Alternatives\.The design above deliberately produces a single space per model, which standard HPO then adapts per dataset\. One could instead generate a separate space for each dataset, or further admit conditional hyperparameters\. We avoid this for the reasons in problem \(iii\), and because public benchmarks are mostly small to medium in size, so exponentially expanding the space with conditional hyperparameters would likely lead to overfitting\. For a practitioner working on a private large application dataset, however, tailoring the space to a specific dataset may prove beneficial\.
## 4Results
### 4\.1Main performance
Figure 2:Aggregate metrics across 45 datasets for tuned individual models \(top\) and greedy ensembles \(bottom\)\. Individual models are tuned for 200 trials on small\-to\-medium datasets and 100 trials on large \(agentic TabICLv2: 100 trials everywhere; default TabICLv2: not tuned\); ensembles are built from pools of 100 \(20 for large datasets\) random configurations\. All metrics are computed over the joint pool; improvement over MLP is the mean relative improvement on the dataset\-native metrics with respect to the tuned base\-space MLP†\. Purple bars and the \(A\) suffix mark the agent\-augmented spaces\. Orange bars are AutoML systems\. AutoGluon \(\+A\) is AutoGluon with the agentic RealMLP and TabM†\. Error bars are95%95\\%confidence intervals across datasets\.Tuned models\.Figure[2](https://arxiv.org/html/2609.16309#S4.F2)\(top\) summarizes the tuned single\-model results: in four of the five model families, the agent\-augmented space improves the aggregate standing over its base space, consistently across all four metrics; the exception is RealMLP, whose base space is meta\-tuned\. Table[4](https://arxiv.org/html/2609.16309#S4.T4)breaks the effect down by task type and dataset size \(see Appendix[B\.2](https://arxiv.org/html/2609.16309#A2.SS2)for the size definition\)\. The gains are cleanest on small\-to\-medium regression datasets: agentic MLP†improves over its base space by\+3\.7%\+3\.7\\%on small and\+1\.6%\+1\.6\\%on medium datasets, agentic LightGBM by\+2\.1%\+2\.1\\%and\+1\.8%\+1\.8\\%, and agentic TabM†by\+1\.8%\+1\.8\\%and\+1\.6%\+1\.6\\%; on large regression datasets the improvements shrink to−0\.4\-0\.4–\+0\.1%\+0\.1\\%\. Classification is more saturated: agentic MLP†still gains\+2\.1%\+2\.1\\%on small data, while on medium datasets the DNNs and LightGBM move within±0\.4%\\pm 0\.4\\%and on large ones the gains stay below\+0\.7%\+0\.7\\%\. TabICLv2 is already the strongest single model on small datasets, leaving little room for improvement \(\+0\.1\+0\.1–\+0\.4%\+0\.4\\%\), but benefits from the agentic space exactly where it lags behind:\+2\.1%\+2\.1\\%on large regression and\+1\.1%\+1\.1\\%on large classification\. RealMLP shows the saturation effect in its strongest form: its base space ships with meta\-learned defaults, and the agentic space leaves the tuned single model essentially unchanged on regression while degrading classification by−0\.8%\-0\.8\\%on medium and large datasets\. The main conclusion is therefore not that an agent always helps, but that agent\-generated spaces help most when the base family exposes useful, composable modules and the task is not already saturated\.
Table 4:Mean relative improvement \(%\) of the agent\-augmented space over the corresponding base HP space, computed on the dataset\-native metrics for tuned single models and greedy ensembles, grouped by task type and dataset size\. Group sizes \(small/medium/large\): regression8/6/68/6/6, classification10/12/310/12/3datasets\. TabICLv2 is not part of the ensembling study\.Tuned and ensembled models\.In Figure[2](https://arxiv.org/html/2609.16309#S4.F2)\(bottom\), ensembling amplifies the effect: ensembles drawn from the agentic spaces occupy the top of the leaderboard, and the agentic RealMLP ensemble is the strongest of all model families, behind only the multi\-model AutoGluon system \(full per\-dataset tables are in Appendix[E\.3](https://arxiv.org/html/2609.16309#A5.SS3)\)\. The ensemble rows of Table[4](https://arxiv.org/html/2609.16309#S4.T4)follow the single\-model pattern but are more uniform across task groups: agentic MLP†gains\+3\.2%\+3\.2\\%on small and\+1\.6%\+1\.6\\%on medium regression and\+2\.1%\+2\.1\\%on small classification, agentic TabM†reaches\+2\.9%\+2\.9\\%on small classification, and RealMLP gains\+0\.9%\+0\.9\\%on small and\+1\.0%\+1\.0\\%on large regression\. Table[5](https://arxiv.org/html/2609.16309#S4.T5)explains where these gains come from\. The agentic pools are only marginally stronger point\-wise: the best single member of an agentic pool matches its base counterpart within\+0\.4%\+0\.4\\%on average\. Their predictions are, however, substantially less correlated\. Hence, the boost from the agentic space is more pronounced for ensembles than for the single models\.
AutoML systems\.Figure[2](https://arxiv.org/html/2609.16309#S4.F2)also compares the search spaces against three AutoML systems run as\-is on the same datasets and test splits: AutoKeras\[[21](https://arxiv.org/html/2609.16309#bib.bib20)\], AutoPyTorch\[[20](https://arxiv.org/html/2609.16309#bib.bib19)\], and AutoGluon\[[19](https://arxiv.org/html/2609.16309#bib.bib13)\]\. Following[Gijsbers et al\. \[47\]](https://arxiv.org/html/2609.16309#bib.bib82)and[Erickson et al\. \[8\]](https://arxiv.org/html/2609.16309#bib.bib11), we set the budget to 4 hours per dataset \(see Appendix[D\.1](https://arxiv.org/html/2609.16309#A4.SS1)for additional details on the settings\)\. AutoKeras and AutoPyTorch search over libraries of human\-implemented neural components and are the natural baselines for our agentic MLP†\. They lag behind it in both the single\-model and the ensemble regime\. One factor is that their libraries leave most of the pipeline fixed: neither modularizes numerical embeddings, target preprocessing, or inference, and categorical embedding has a single component in AutoPyTorch and none in AutoKeras\. Their4444and44interchangeable components sit mostly on preprocessing and the optimizer, against5757–6161spread over all axes for the agentic MLP†space \(Appendix[D\.1](https://arxiv.org/html/2609.16309#A4.SS1)\)\. For AutoGluon we use thebest\_qualitypreset and pass it both the train and validation sets\. AutoGluon fits a meta\-learned portfolio of110110fixed configurations across seven conventional model families \(LightGBM, CatBoost, XGBoost, random forest, extra trees, and two MLP variants; Appendix[D\.1](https://arxiv.org/html/2609.16309#A4.SS1)\),88\-fold bags each one, feeds the out\-of\-fold predictions to a second stack layer of the same families, and greedily weights the final predictions\. This design is complementary to our approach, and we test the combination directly: AutoGluon \(\+A\) receives 8 randomly\-sampled configurations of our agentic RealMLP and TabM†models\. The augmented system outperforms the base\. AutoGluon selects the agent\-implemented models into its final weighted ensemble on4343of the4545datasets, assigning them54%54\\%of the ensemble weight on average\.
TabArena\.Finally, the conclusions transfer to the recent TabArena benchmark \(51 datasets, official protocol; see Appendix[A](https://arxiv.org/html/2609.16309#A1)\)\. With tuning and ensembling, the agentic spaces improve the official Elo scores of MLP†, TabM, RealMLP, and TabICLv2, placing both the agentic TabICLv2 and the agentic RealMLP above AutoGluon with the samebest\_qualityportfolio of conventional models as above\. The agentic RealMLP thus overtakes AutoGluon on TabArena but not in Figure[2](https://arxiv.org/html/2609.16309#S4.F2)\. The likely reason is the protocol\. Under TabArena, every configuration is itself an 8\-fold cross\-validated ensemble, i\.e\., the same bagging machinery that AutoGluon uses internally\. Its greedy ensemble draws on all200200tuning configurations, which is possible since they are randomly sampled\. In our benchmark, each configuration is fit once on the training subset, scored once on the validation subset, and yields a single set of test predictions\. Our greedy ensemble draws on100100fresh random configurations \(2020on large datasets\), because the TPE trials of the single\-model study are highly correlated and cannot be reused\. Both choices are deliberate: TPE strengthens the individual models, and forgoing cross\-validation admits larger datasets as well as datasets with grouped and temporal splits, for which cross\-validation is not well defined\. Figure[2](https://arxiv.org/html/2609.16309#S4.F2)therefore gives a more conservative view of the individual model families than TabArena does\.
Table 5:Greedy\-ensemble diversity aggregated over all 45 datasets\.ρ\\rhois the mean pairwise test\-prediction correlation among the selected members \(lower→\\rightarrowmore diverse pool\);Δbase\\Delta\_\{\\mathrm\{base\}\}is the mean relative improvement of the agentic space over the base space on the dataset\-native metrics, reported for the pool’s strongest single member and for the greedy ensemble built from the same pool\.
### 4\.2Agent cost and amortization
Table 6:Agent resource usage per model family\. The columns report output tokens and wall\-clock generation time for the main agent \(Claude Code\) and the ablation agent \(Codex\); the agent is run once per model family, and token counts are reported instead of dollar cost because provider pricing changes over time\.In Table[6](https://arxiv.org/html/2609.16309#S4.T6), we report the agent’s output token usage \(LLM thinking and generation\) and wall\-clock generation time for Claude Code Opus 4\.8 and Codex GPT 5\.5\. We omit input and cached tokens, which are dominated by the fixed system prompt, tool signatures, and environment setup shipped with the agent\. Given current LLM costs per token\[[48](https://arxiv.org/html/2609.16309#bib.bib34),[49](https://arxiv.org/html/2609.16309#bib.bib35)\], we estimate that a single hypothesis generation loop costs approximately1010USD\. The agent is run once per model family, while HPO is run per dataset\. This distinction is important: agent generation takes roughly3535–100100minutes depending on model family and agent, but the cost is amortized across all later datasets unlike the autoresearch\[[30](https://arxiv.org/html/2609.16309#bib.bib2)\]\. The method is therefore most attractive when the same generated space will be reused across many datasets\.
## 5Analysis
### 5\.1Which modules matter?
Figure 3:*Top:*Module importance from fANOVA for MLP, normalized to sum to one per dataset\. Green boxes are the code modules with agent\-generated candidate implementations; blue boxes are conventional numerical hyperparameters\.*Bottom:*Selection lift of the candidate implementations of the four most important modules\. For each candidate,nnis the number of datasets \(out of 45\) on which it appears in the best HPO trial, and the lift isnndivided by the count expected if the winner were drawn uniformly among the module’s candidates; bars above one thus mark candidates selected more often than the uniform baseline\.From the fANOVA chart of Figure[3](https://arxiv.org/html/2609.16309#S5.F3), we see substantial importance is assigned to several modules extended by the agent: numerical embeddings, architecture, training, and numerical preprocessing\. Further, the per\-candidate selection\-lift plot of Figure[3](https://arxiv.org/html/2609.16309#S5.F3)suggests that no single candidate dominates: the selections are spread across each module’s candidate pool with no clean winner\.
Aggregating the winning HPO trials across the 18 MLP regression datasets, we observe that the selected candidate combinations are pairwise distinct, and every candidate except two training\-side ones appears in at least one of them\. This suggests that the joint selection is necessary\. Which candidates win is thus largely dataset\-specific: the agent contributes a reusable basis of implementations, and the per\-dataset composition has to be found by HPO\. The candidate\-pool ablation in Section[5\.3](https://arxiv.org/html/2609.16309#S5.SS3)is consistent with this picture: no subsampled pool recovers the full space’s performance\.
Table 7:Stability of agent\-generated MLP†search spaces across five independent re\-generations per agent, evaluated on all 45 datasets\. Mean rank, normalized score, and Elo \(anchored at the base MLP†= 1000\) are computed in the joint pool of the 12 spaces \(base, released, and the ten re\-generations\); improvement over MLP†is the mean relative improvement on the dataset\-native metrics\. Agent rows report the mean±\\pmhalf\-width of a95%95\\%tt\-interval over the five re\-generations; per\-run values are in Table[22](https://arxiv.org/html/2609.16309#A5.T22)\.
### 5\.2Agent choice and stability
Table[7](https://arxiv.org/html/2609.16309#S5.T7)compares agent\-generated MLP†search spaces across five independent re\-generations for Claude Code with Opus 4\.8 atmaxthinking level and Codex with GPT\-5\.5 atxhighthinking level\[[35](https://arxiv.org/html/2609.16309#bib.bib75),[34](https://arxiv.org/html/2609.16309#bib.bib71),[36](https://arxiv.org/html/2609.16309#bib.bib77),[37](https://arxiv.org/html/2609.16309#bib.bib76)\], evaluated on all 45 datasets\. Every re\-generated space improves over the base MLP†space on all four aggregate metrics, and the released space used in the main experiments lies within the re\-generation spread\. The two agents perform on par: Codex is somewhat more consistent \(Elo1141±221141\\pm 22vs\.1132±611132\\pm 61\), while the best and the worst single runs both come from Claude \(Elo11821182and10561056\)\. Per\-run results are given in Table[22](https://arxiv.org/html/2609.16309#A5.T22)in Appendix[E\.1](https://arxiv.org/html/2609.16309#A5.SS1)\.
### 5\.3Candidate\-pool size
In Section[5\.1](https://arxiv.org/html/2609.16309#S5.SS1), we observe that the selected candidate combinations are pairwise distinct across datasets, which suggests that growing the search space should help\. However, beyond some size, a larger search space could exacerbate overfitting to the validation subset\. We check both claims by re\-fitting the agentic MLP†over progressively subsampled candidate pools: for each size in\{8,16,32,48\}\\\{8,16,32,48\\\}we draw five random subsets of the candidates from the full pool \(6161for regression,5757for classification\), rebuild the search space, and re\-run the full 200\-trial HPO on 18 randomly chosen datasets\. In Table[8](https://arxiv.org/html/2609.16309#S5.T8), all four aggregate metrics improve monotonically with the pool size, and the full pool remains clearly the best \(Elo11431143vs\.10861086for the largest subsets\): growing the search space indeed helps\. We also observe no downturn at the current pool sizes, so overfitting does not yet set in at this scale\.
Table 8:Candidate\-pool subsampling for the agentic MLP†space on 18 randomly chosen datasets\. For each pool size we draw five random subsets of the agent\-generated candidate implementations, rebuild the search space, and re\-run the full 200\-trial HPO; the last row uses the complete pool\. Metrics follow the main\-figure definitions and are computed per replicate in a pool of six spaces \(base, one random subset per size, full\), with Elo anchored at the base MLP†= 1000; each value is the mean±\\pmhalf\-width of a95%95\\%tt\-interval over the five replicates, so the intervals measure the sensitivity to which candidates are drawn\. Base and full are the same space in every replicate; their small intervals reflect only the changing comparison pool\.
### 5\.4Trial budget\.
The fixed trial\-budget comparison of Figure[2](https://arxiv.org/html/2609.16309#S4.F2)could in principle depend on the chosen trial budget: a larger space needs more trials to be explored, so agentic spaces might only win once the budget is large enough\. Figure[4](https://arxiv.org/html/2609.16309#S5.F4)therefore reports the average improvement of the agentic space over the base space at each matched Optuna budget\. On small\-and\-medium datasets the gap is positive at every budget for MLP†, TabM†, and LightGBM, while the agentic RealMLP stays within±0\.4%\\pm 0\.4\\%of its base space\. On large datasets the gaps are smaller and mixed, with the exception of TabICLv2 \(\+1\.8%\+1\.8\\%\)\. The gaps change little from5050trials onwards, so the conclusions of Section[4\.1](https://arxiv.org/html/2609.16309#S4.SS1)are not an artifact of a particular budget value\.
Figure 4:Effect of the matched Optuna trial budget for small\-and\-medium \(left\) and large \(right\) datasets\. The cell color and value give the mean relative improvement of the agentic space over the corresponding base space at matched budget, computed on the dataset\-native metrics \(%\); green marks budgets where the agentic space is ahead\. For TabICLv2, the cells do not vary with the budget, because the default TabICLv2 is not tuned and the agentic TabICLv2 is tuned at a fixed100100trials on every dataset\.
## 6Limitations
The method expands a finite\-budget HPO problem, so the superset relation𝒮HPO⊆𝒮agent\\mathcal\{S\}\_\{\\mathrm\{HPO\}\}\\subseteq\\mathcal\{S\}\_\{\\mathrm\{agent\}\}does not guarantee finite\-budget improvement\. Larger spaces can overfit validation\. The agentic spaces can also raise the per\-fit cost, since the agent may propose more expensive optimizers or test\-time augmentations: the median fit\-time ratio to the base space ranged from0\.9×0\.9\\times\(RealMLP\) to2\.4×2\.4\\times\(TabICLv2\) in our runs\. Agent\-generated code may potentially contain bugs; hence, it must be manually inspected before shipping\. A potential improvement to the evaluation and HPO pipeline is to use cross\-validation; however, this can be prohibitively expensive in practice\. The approach also depends on the quality and reproducibility of the agent\-generated candidate sets\. Agent products, prompts, and retrieval corpora change over time, so they must be logged as experimental artifacts\. We evaluate five model families and 45 datasets, which is broad enough to show the main pattern but not enough to claim universal gains across all tabular regimes\. Our ensembling study covers four model families and reports a single greedy\-ensembling run per \(dataset, search space\) over random candidate pools; extending it to foundation models, and to pools drawn from actual HPO trajectories, is left for future work\. Finally, we intentionally avoid dataset\-specific prompting during generation\. This protects against leakage and amortizes cost, but it can miss modules that would be useful for a particular dataset’s semantics or feature types\. We detail some of the future research directions in Appendix[C](https://arxiv.org/html/2609.16309#A3)\.
## 7Conclusion
We propose to use LLM agents to generate structured, executable module implementations that expand the search space for HPO\. By pairing this generated basis with a classical HPO algorithm, we obtain consistent gains for classic DNNs, while for GBDT, foundational, and heavily meta\-tuned models the gains are smaller and concentrated in the regimes where the base model is not already saturated\. The module and combination analyses suggest that the value comes from dataset\-specific composition: no single generated candidate is universally best, but the candidate basis gives HPO more useful ways to adapt the model pipeline\. This makes agentic search\-space generation a practical middle ground between fixed HPO grids and fully autonomous research loops\. We also find that agent\-produced blocks add diversity that boosts ensemble performance: the greedy ensemble over the agentic RealMLP space is the strongest of all model families in our evaluation, ahead of a tuned tabular foundation model and behind only the multi\-model AutoGluon system\.
## References
- \[1\]D\. Holzmüller, L\. Grinsztajn, and I\. Steinwart\(2024\)Better by default: strong pre\-tuned mlps and boosted trees on tabular data\.InAdvances in Neural Information Processing Systems,Cited by:[§B\.4](https://arxiv.org/html/2609.16309#A2.SS4.SSS0.Px3.p1.1),[Appendix C](https://arxiv.org/html/2609.16309#A3.p1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3),[§1](https://arxiv.org/html/2609.16309#S1.p1.1),[§2](https://arxiv.org/html/2609.16309#S2.p2.1),[§3\.1](https://arxiv.org/html/2609.16309#S3.SS1.p3.1)\.
- \[2\]Y\. Gorishniy, A\. Kotelnikov, and A\. Babenko\(2025\)TabM: advancing tabular deep learning with parameter\-efficient ensembling\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§B\.2](https://arxiv.org/html/2609.16309#A2.SS2.p1.1),[§B\.4](https://arxiv.org/html/2609.16309#A2.SS4.SSS0.Px1.p1.1),[§B\.4](https://arxiv.org/html/2609.16309#A2.SS4.SSS0.Px2.p1.1),[Appendix C](https://arxiv.org/html/2609.16309#A3.p1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p5.1.5.2.1.1),[§1](https://arxiv.org/html/2609.16309#S1.p1.1),[§2](https://arxiv.org/html/2609.16309#S2.p1.1),[§2](https://arxiv.org/html/2609.16309#S2.p2.1),[§3\.1](https://arxiv.org/html/2609.16309#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2609.16309#S3.SS1.p3.1)\.
- \[3\]H\. Ye, H\. Yin, D\. Zhan, and W\. Chao\(2025\)Revisiting nearest neighbor for tabular data: a deep tabular baseline two decades later\.InInternational Conference on Learning Representations \(ICLR\),Note:ModernNCACited by:[§1](https://arxiv.org/html/2609.16309#S1.p1.1),[§2](https://arxiv.org/html/2609.16309#S2.p1.1)\.
- \[4\]N\. Hollmann, S\. Müller, K\. Eggensperger, and F\. Hutter\(2023\)TabPFN: a transformer that solves small tabular classification problems in a second\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2609.16309#S1.p1.1),[§2](https://arxiv.org/html/2609.16309#S2.p1.1)\.
- \[5\]J\. Qu, D\. Holzmüller, G\. Varoquaux, and M\. Le Morvan\(2025\)TabICL: a tabular foundation model for in\-context learning on large data\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p7.1.3.2.1.1),[§1](https://arxiv.org/html/2609.16309#S1.p1.1)\.
- \[6\]Zhanget al\.\(2025\)LimiX: unleashing structured\-data modeling capability for generalist intelligence\.arXiv preprint arXiv:2509\.03505\.Cited by:[§1](https://arxiv.org/html/2609.16309#S1.p1.1),[§2](https://arxiv.org/html/2609.16309#S2.p1.1),[§2](https://arxiv.org/html/2609.16309#S2.p2.1)\.
- \[7\]I\. Rubachev, N\. Kartashev, Y\. Gorishniy, and A\. Babenko\(2025\)TabReD: analyzing pitfalls and filling the gaps in tabular deep learning benchmarks\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§B\.2](https://arxiv.org/html/2609.16309#A2.SS2.p1.1),[§1](https://arxiv.org/html/2609.16309#S1.p1.1),[§2](https://arxiv.org/html/2609.16309#S2.p1.1),[§3\.1](https://arxiv.org/html/2609.16309#S3.SS1.p1.1)\.
- \[8\]N\. Erickson, L\. Purucker, A\. Tschalzev, D\. Holzmüller, P\. M\. Desai, D\. Salinas, and F\. Hutter\(2025\)TabArena: a living benchmark for machine learning on tabular data\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,Cited by:[Appendix A](https://arxiv.org/html/2609.16309#A1.p1.1),[§B\.2](https://arxiv.org/html/2609.16309#A2.SS2.p1.1),[§B\.3](https://arxiv.org/html/2609.16309#A2.SS3.p1.1),[§D\.1](https://arxiv.org/html/2609.16309#A4.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2609.16309#S1.p1.1),[§2](https://arxiv.org/html/2609.16309#S2.p1.1),[§3\.1](https://arxiv.org/html/2609.16309#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2609.16309#S3.SS1.p3.1),[§4\.1](https://arxiv.org/html/2609.16309#S4.SS1.p3.1)\.
- \[9\]S\. Hegselmann, A\. Buendia, H\. Lang, M\. Agrawal, X\. Jiang, and D\. Sontag\(2023\)TabLLM: few\-shot classification of tabular data with large language models\.InProceedings of the 26th International Conference on Artificial Intelligence and Statistics \(AISTATS\),Cited by:[§1](https://arxiv.org/html/2609.16309#S1.p2.1),[§2](https://arxiv.org/html/2609.16309#S2.p3.1)\.
- \[10\]J\. Gardner, J\. C\. Perdomo, and L\. Schmidt\(2024\)Large scale transfer learning for tabular data via language modeling\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2609.16309#S1.p2.1),[§2](https://arxiv.org/html/2609.16309#S2.p3.1)\.
- \[11\]N\. Hollmann, S\. Müller, and F\. Hutter\(2023\)Large language models for automated data science: introducing CAAFE for context\-aware automated feature engineering\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2609.16309#S1.p2.1),[§2](https://arxiv.org/html/2609.16309#S2.p3.1)\.
- \[12\]S\. Han, J\. Yoon, S\. Ö\. Arik, and T\. Pfister\(2024\)Large language models can automatically engineer features for few\-shot tabular learning\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2609.16309#S1.p2.1),[§2](https://arxiv.org/html/2609.16309#S2.p3.1)\.
- \[13\]L\. Grinsztajn, M\. J\. Kim, E\. Oyallon, and G\. Varoquaux\(2023\)Modeling string entries for tabular data prediction: do we need big large language models?\.InNeurIPS 2023 Second Table Representation Learning Workshop,Cited by:[§1](https://arxiv.org/html/2609.16309#S1.p2.1)\.
- \[14\]G\. Yakushev, A\. Shutova, I\. Rubachev, R\. Sergazinov, and A\. BabenkoTalking trees: reasoning\-assisted induction of decision trees for tabular data\.InEurIPS 2025 Workshop: AI for Tabular Data,Cited by:[§1](https://arxiv.org/html/2609.16309#S1.p2.1)\.
- \[15\]Y\. Gorishniy, I\. Rubachev, N\. Kartashev, D\. Shlenskii, A\. Kotelnikov, and A\. Babenko\(2024\)TabR: tabular deep learning meets nearest neighbors\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§B\.4](https://arxiv.org/html/2609.16309#A2.SS4.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2609.16309#S2.p1.1)\.
- \[16\]J\. Qu, D\. Holzmüller, G\. Varoquaux, and M\. Le Morvan\(2026\)TabICLv2: a better, faster, scalable, and open tabular foundation model\.arXiv preprint arXiv:2602\.11139\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2602.11139),[Link](https://arxiv.org/abs/2602.11139)Cited by:[§B\.4](https://arxiv.org/html/2609.16309#A2.SS4.SSS0.Px5.p1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p7.1.4.2.1.1),[§2](https://arxiv.org/html/2609.16309#S2.p1.1),[§2](https://arxiv.org/html/2609.16309#S2.p2.1),[§3\.1](https://arxiv.org/html/2609.16309#S3.SS1.p3.1)\.
- \[17\]M\. Feurer, A\. Klein, K\. Eggensperger, J\. Springenberg, M\. Blum, and F\. Hutter\(2015\)Efficient and robust automated machine learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§D\.1](https://arxiv.org/html/2609.16309#A4.SS1.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2609.16309#S2.p2.1)\.
- \[18\]R\. Caruana, A\. Niculescu\-Mizil, G\. Crew, and A\. Ksikes\(2004\)Ensemble selection from libraries of models\.InProceedings of the International Conference on Machine Learning,External Links:[Link](https://www.cs.cornell.edu/~alexn/papers/shotgun.icml04.revised.rev2.pdf)Cited by:[Appendix A](https://arxiv.org/html/2609.16309#A1.p1.1),[§B\.5](https://arxiv.org/html/2609.16309#A2.SS5.SSS0.Px2.p1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p7.1.5.2.1.1),[§E\.3](https://arxiv.org/html/2609.16309#A5.SS3.p1.1),[§2](https://arxiv.org/html/2609.16309#S2.p2.1),[§3\.1](https://arxiv.org/html/2609.16309#S3.SS1.p4.1)\.
- \[19\]N\. Erickson, J\. Mueller, A\. Shirkov, H\. Zhang, P\. Larroy, M\. Li, and A\. Smola\(2020\)AutoGluon\-tabular: robust and accurate AutoML for structured data\.arXiv preprint arXiv:2003\.06505\.Cited by:[§2](https://arxiv.org/html/2609.16309#S2.p2.1),[§4\.1](https://arxiv.org/html/2609.16309#S4.SS1.p3.1)\.
- \[20\]L\. Zimmer, M\. Lindauer, and F\. Hutter\(2021\)Auto\-PyTorch: multi\-fidelity metalearning for efficient and robust AutoDL\.IEEE Transactions on Pattern Analysis and Machine Intelligence43\(9\),pp\. 3079–3090\.Cited by:[§2](https://arxiv.org/html/2609.16309#S2.p2.1),[§4\.1](https://arxiv.org/html/2609.16309#S4.SS1.p3.1)\.
- \[21\]H\. Jin, Q\. Song, and X\. Hu\(2019\)Auto\-Keras: an efficient neural architecture search system\.InProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pp\. 1946–1956\.Cited by:[§2](https://arxiv.org/html/2609.16309#S2.p2.1),[§4\.1](https://arxiv.org/html/2609.16309#S4.SS1.p3.1)\.
- \[22\]B\. Zoph and Q\. V\. Le\(2017\)Neural architecture search with reinforcement learning\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.16309#S2.p2.1)\.
- \[23\]H\. Liu, K\. Simonyan, and Y\. Yang\(2019\)DARTS: differentiable architecture search\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.16309#S2.p2.1)\.
- \[24\]T\. Liu, N\. Astorga, N\. Seedat, and M\. van der Schaar\(2024\)Large language models to enhance bayesian optimization\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.16309#S2.p4.1)\.
- \[25\]Y\. Chen, X\. Song, C\. Lee, Z\. Wang, Q\. Zhang, D\. Dohan, K\. Kawakami, G\. Kochanski, A\. Doucet, M\. Ranzato, S\. Perel, and N\. de Freitas\(2022\)Towards learning universal hyperparameter optimizers with transformers\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2609.16309#S2.p4.1)\.
- \[26\]J\. Meindl, Y\. Tian, T\. Cui, V\. Thost, Z\. Hong, J\. Chen, W\. Matusik, and M\. Konaković Luković\(2025\)GPTOpt: towards efficient LLM\-based black\-box optimization\.arXiv preprint arXiv:2510\.25404\.Cited by:[§2](https://arxiv.org/html/2609.16309#S2.p4.1)\.
- \[27\]N\. Menet, A\. Terzić, M\. Hersche, A\. Krause, and A\. Rahimi\(2025\)Thompson sampling via fine\-tuning of LLMs\.arXiv preprint arXiv:2510\.13328\.Cited by:[§2](https://arxiv.org/html/2609.16309#S2.p4.1)\.
- \[28\]R\. Tan, M\. Chen, K\. Xue, Y\. Wang, Y\. Wang, S\. Fu, and C\. Qian\(2025\)Towards universal offline black\-box optimization via learning language model embeddings\.InInternational Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2609.16309#S2.p4.1)\.
- \[29\]C\. Lu, C\. Lu, R\. T\. Lange, J\. Foerster, J\. Clune, and D\. Ha\(2024\)The AI scientist: towards fully automated open\-ended scientific discovery\.External Links:2408\.06292Cited by:[§2](https://arxiv.org/html/2609.16309#S2.p4.1)\.
- \[30\]A\. Karpathy\(2026\)Autoresearch: ai agents running research loops\.GitHub\.Note:[https://github\.com/karpathy/autoresearch](https://github.com/karpathy/autoresearch)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p1.1),[§2](https://arxiv.org/html/2609.16309#S2.p4.1),[§3\.2](https://arxiv.org/html/2609.16309#S3.SS2.p1.1),[Table 3](https://arxiv.org/html/2609.16309#S3.T3),[Table 3](https://arxiv.org/html/2609.16309#S3.T3.2.2.1),[§4\.2](https://arxiv.org/html/2609.16309#S4.SS2.p1.1)\.
- \[31\]F\. Ferreira, L\. Wobbe, A\. Krishnakumar, F\. Hutter, and A\. Zela\(2026\)Can llms beat classical hyperparameter optimization algorithms? a study on autoresearch\.arXiv preprint arXiv:2603\.24647\.Cited by:[§2](https://arxiv.org/html/2609.16309#S2.p4.1),[§3\.2](https://arxiv.org/html/2609.16309#S3.SS2.p4.1)\.
- \[32\]Y\. Gorishniy, I\. Rubachev, and A\. Babenko\(2022\)On embeddings for numerical features in tabular deep learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§B\.4](https://arxiv.org/html/2609.16309#A2.SS4.SSS0.Px1.p1.1),[Appendix C](https://arxiv.org/html/2609.16309#A3.p1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.5.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.5.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p5.1.4.2.1.1),[§3\.1](https://arxiv.org/html/2609.16309#S3.SS1.p3.1),[Table 2](https://arxiv.org/html/2609.16309#S3.T2.10.2.3.1.1)\.
- \[33\]G\. Ke, Q\. Meng, T\. Finley, T\. Wang, W\. Chen, W\. Ma, Q\. Ye, and T\. Liu\(2017\)LightGBM: a highly efficient gradient boosting decision tree\.InAdvances in Neural Information Processing Systems,Vol\.30,pp\. 3146–3154\.Cited by:[§B\.4](https://arxiv.org/html/2609.16309#A2.SS4.SSS0.Px4.p1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3),[§3\.1](https://arxiv.org/html/2609.16309#S3.SS1.p3.1)\.
- \[34\]Anthropic\(2026\)Claude code\.Note:[https://www\.anthropic\.com/product/claude\-code](https://www.anthropic.com/product/claude-code)Accessed: 2026\-05\-07Cited by:[§3\.1](https://arxiv.org/html/2609.16309#S3.SS1.p3.1),[§5\.2](https://arxiv.org/html/2609.16309#S5.SS2.p1.1)\.
- \[35\]Anthropic\(2026\)Claude Opus 4\.8\.Note:[https://www\.anthropic\.com/news/claude\-opus\-4\-8](https://www.anthropic.com/news/claude-opus-4-8)Large language model\. Accessed 2026\-06\-26Cited by:[§3\.1](https://arxiv.org/html/2609.16309#S3.SS1.p3.1),[§5\.2](https://arxiv.org/html/2609.16309#S5.SS2.p1.1)\.
- \[36\]OpenAI\(2025\)Introducing Codex\.Note:[https://openai\.com/index/introducing\-codex/](https://openai.com/index/introducing-codex/)AI coding agent\. Accessed 2026\-06\-26Cited by:[§3\.1](https://arxiv.org/html/2609.16309#S3.SS1.p3.1),[§5\.2](https://arxiv.org/html/2609.16309#S5.SS2.p1.1)\.
- \[37\]OpenAI\(2026\)Introducing GPT\-5\.5\.Note:[https://openai\.com/index/introducing\-gpt\-5\-5/](https://openai.com/index/introducing-gpt-5-5/)Large language model\. Accessed 2026\-06\-26Cited by:[§3\.1](https://arxiv.org/html/2609.16309#S3.SS1.p3.1),[§5\.2](https://arxiv.org/html/2609.16309#S5.SS2.p1.1)\.
- \[38\]J\. Bergstra, R\. Bardenet, Y\. Bengio, and B\. Kégl\(2011\)Algorithms for hyper\-parameter optimization\.InAdvances in Neural Information Processing Systems,Cited by:[§B\.5](https://arxiv.org/html/2609.16309#A2.SS5.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2609.16309#S3.SS1.p4.1)\.
- \[39\]T\. Akiba, S\. Sano, T\. Yanase, T\. Ohta, and M\. Koyama\(2019\)Optuna: a next\-generation hyperparameter optimization framework\.InProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pp\. 2623–2631\.External Links:[Document](https://dx.doi.org/10.1145/3292500.3330701)Cited by:[§B\.5](https://arxiv.org/html/2609.16309#A2.SS5.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2609.16309#S3.SS1.p4.1)\.
- \[40\]B\. T\. Polyak and A\. B\. Juditsky\(1992\)Acceleration of stochastic approximation by averaging\.SIAM Journal on Control and Optimization30\(4\),pp\. 838–855\.External Links:[Document](https://dx.doi.org/10.1137/0330046)Cited by:[§D\.2](https://arxiv.org/html/2609.16309#A4.SS2.SSS0.Px3.p2.2),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.8.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.8.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p5.1.7.2.1.1),[Table 2](https://arxiv.org/html/2609.16309#S3.T2.10.2.4.1.1)\.
- \[41\]A\. Krishnamurthy, K\. Harris, D\. J\. Foster, C\. Zhang, and A\. Slivkins\(2024\)Can large language models explore in\-context?\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§3\.2](https://arxiv.org/html/2609.16309#S3.SS2.p3.1)\.
- \[42\]Y\. Wang, J\. Hao, Y\. Shi, K\. Yuan, and M\. Sun\(2026\)No time like the present: agentic test\-time training for llm agents\.arXiv preprint arXiv:2607\.03441\.Cited by:[§3\.2](https://arxiv.org/html/2609.16309#S3.SS2.p3.1)\.
- \[43\]D\. Xu, H\. Wu, W\. Shi, Y\. Cui, Y\. Liu, J\. Li, H\. Ma, A\. Liu, J\. Zhu, and J\. Xu\(2026\)LoopGuard: breaking self\-reinforcing attention loops via dynamic kv cache intervention\.arXiv preprint arXiv:2604\.10044\.Cited by:[§3\.2](https://arxiv.org/html/2609.16309#S3.SS2.p3.1)\.
- \[44\]I\. Loshchilov and F\. Hutter\(2019\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[§3\.2](https://arxiv.org/html/2609.16309#S3.SS2.p7.1)\.
- \[45\]I\. Yeo and R\. A\. Johnson\(2000\)A new family of power transformations to improve normality or symmetry\.Biometrika87\(4\),pp\. 954–959\.External Links:[Document](https://dx.doi.org/10.1093/biomet/87.4.954)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.2.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.2.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p5.1.2.2.1.1),[§3\.2](https://arxiv.org/html/2609.16309#S3.SS2.p7.1)\.
- \[46\]X\. Chen, C\. Liang, D\. Huang, E\. Real, K\. Wang, H\. Pham, X\. Dong, T\. Luong, C\. Hsieh, Y\. Lu,et al\.\(2023\)Symbolic discovery of optimization algorithms\.Advances in neural information processing systems36,pp\. 49205–49233\.Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.9.2.1.1),[§3\.2](https://arxiv.org/html/2609.16309#S3.SS2.p7.1)\.
- \[47\]P\. Gijsbers, M\. L\. P\. Bueno, S\. Coors, E\. LeDell, S\. Poirier, J\. Thomas, B\. Bischl, and J\. Vanschoren\(2024\)AMLB: an AutoML benchmark\.Journal of Machine Learning Research25\(101\),pp\. 1–65\.Cited by:[§D\.1](https://arxiv.org/html/2609.16309#A4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.16309#S4.SS1.p3.1)\.
- \[48\]Anthropic\(2026\)Claude API Pricing\.Note:[https://docs\.anthropic\.com/en/docs/about\-claude/pricing](https://docs.anthropic.com/en/docs/about-claude/pricing)Accessed: 2026\-05\-06\. Lists Claude Opus 4\.7 pricing as $5 per million input tokens and $25 per million output tokens\.Cited by:[§4\.2](https://arxiv.org/html/2609.16309#S4.SS2.p1.1)\.
- \[49\]OpenAI\(2026\)API Pricing\.Note:[https://openai\.com/api/pricing/](https://openai.com/api/pricing/)Accessed: 2026\-05\-06\. Lists GPT\-5\.5 pricing as $5 per million input tokens, $0\.50 per million cached input tokens, and $30 per million output tokens\.Cited by:[§4\.2](https://arxiv.org/html/2609.16309#S4.SS2.p1.1)\.
- \[50\]D\. Salinas and N\. Erickson\(2024\)Tabrepo: a large scale repository of tabular model evaluations and its automl applications\.AutoML Conference \(ABCD Track\)\.Cited by:[3rd item](https://arxiv.org/html/2609.16309#A2.I1.i3.p1.1)\.
- \[51\]W\. Chiang, L\. Zheng, Y\. Sheng, A\. N\. Angelopoulos, T\. Li, D\. Li, H\. Zhang, B\. Zhu, M\. Jordan, J\. E\. Gonzalez,et al\.\(2024\)Chatbot arena: an open platform for evaluating llms by human preference\.arXiv preprint arXiv:2403\.04132\.Cited by:[4th item](https://arxiv.org/html/2609.16309#A2.I1.i4.p1.1)\.
- \[52\]J\. TangTPE – how many startup trials are enough?\.Note:Kaggle NotebookAccessed: 2026\-07\-09External Links:[Link](https://www.kaggle.com/code/jerrytang88/tpe-how-many-startup-trials-are-enough)Cited by:[§B\.5](https://arxiv.org/html/2609.16309#A2.SS5.SSS0.Px1.p1.1)\.
- \[53\]Y\. Gorishniy, I\. Rubachev, D\. Feoktistov, and A\. Babenko\(2026\)Benchmarking optimizers for mlps in tabular deep learning\.arXiv preprint arXiv:2604\.15297\.Cited by:[Appendix C](https://arxiv.org/html/2609.16309#A3.p1.1)\.
- \[54\]K\. Weinberger, A\. Dasgupta, J\. Langford, A\. Smola, and J\. Attenberg\(2009\)Feature hashing for large scale multitask learning\.InProceedings of the International Conference on Machine Learning,pp\. 1113–1120\.External Links:[Document](https://dx.doi.org/10.1145/1553374.1553516)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.3.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.6.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.3.2.1.1)\.
- \[55\]D\. Micci\-Barreca\(2001\)A preprocessing scheme for high\-cardinality categorical attributes in classification and prediction problems\.ACM SIGKDD Explorations Newsletter3\(1\),pp\. 27–32\.External Links:[Document](https://dx.doi.org/10.1145/507533.507538)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.3.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.6.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p6.1.2.2.1.1)\.
- \[56\]A\. Rahimi and B\. Recht\(2007\)Random features for large\-scale kernel machines\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://papers.nips.cc/paper/3182-random-features-for-large-scale-kernel-machines)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.5.2.1.1)\.
- \[57\]C\. Guo and F\. Berkhahn\(2016\)Entity embeddings of categorical variables\.arXiv preprint arXiv:1604\.06737\.External Links:[Link](https://arxiv.org/abs/1604.06737)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.6.2.1.1)\.
- \[58\]Y\. N\. Dauphin, A\. Fan, M\. Auli, and D\. Grangier\(2017\)Language modeling with gated convolutional networks\.InProceedings of the International Conference on Machine Learning,pp\. 933–941\.External Links:[Link](https://proceedings.mlr.press/v70/dauphin17a.html)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.7.2.1.1)\.
- \[59\]N\. Shazeer\(2020\)GLU variants improve transformer\.arXiv preprint arXiv:2002\.05202\.External Links:[Link](https://arxiv.org/abs/2002.05202)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.7.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.7.2.1.1)\.
- \[60\]K\. He, X\. Zhang, S\. Ren, and J\. Sun\(2016\)Deep residual learning for image recognition\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 770–778\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2016.90)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.7.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p5.1.6.2.1.1)\.
- \[61\]J\. Hu, L\. Shen, and G\. Sun\(2018\)Squeeze\-and\-excitation networks\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 7132–7141\.External Links:[Link](https://openaccess.thecvf.com/content_cvpr_2018/html/Hu_Squeeze-and-Excitation_Networks_CVPR_2018_paper.html)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.7.2.1.1)\.
- \[62\]I\. Loshchilov and F\. Hutter\(2017\)SGDR: stochastic gradient descent with warm restarts\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/1608.03983)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.8.2.1.1)\.
- \[63\]P\. Izmailov, D\. Podoprikhin, T\. Garipov, D\. Vetrov, and A\. G\. Wilson\(2018\)Averaging weights leads to wider optima and better generalization\.InConference on Uncertainty in Artificial Intelligence,External Links:[Link](https://arxiv.org/abs/1803.05407)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.8.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.8.2.1.1)\.
- \[64\]H\. Zhang, M\. Cisse, Y\. N\. Dauphin, and D\. Lopez\-Paz\(2018\)Mixup: beyond empirical risk minimization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=r1Ddp1-Rb)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.8.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.8.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p5.1.7.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p6.1.3.2.1.1)\.
- \[65\]M\. R\. Zhang, J\. Lucas, G\. Hinton, and J\. Ba\(2019\)Lookahead optimizer:kksteps forward, 1 step back\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://papers.nips.cc/paper/2019/hash/90fd4f88f588ae64038134f1eeaa023f-Abstract.html)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.9.2.1.1)\.
- \[66\]P\. J\. Huber\(1964\)Robust estimation of a location parameter\.The Annals of Mathematical Statistics35\(1\),pp\. 73–101\.External Links:[Document](https://dx.doi.org/10.1214/aoms/1177703732)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.10.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.10.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p5.1.9.2.1.1)\.
- \[67\]C\. Szegedy, V\. Vanhoucke, S\. Ioffe, J\. Shlens, and Z\. Wojna\(2016\)Rethinking the inception architecture for computer vision\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 2818–2826\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2016.308)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.10.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p6.1.4.2.1.1)\.
- \[68\]Y\. Gal and Z\. Ghahramani\(2016\)Dropout as a bayesian approximation: representing model uncertainty in deep learning\.InProceedings of the International Conference on Machine Learning,pp\. 1050–1059\.External Links:[Link](https://proceedings.mlr.press/v48/gal16.html)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.11.2.1.1)\.
- \[69\]C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger\(2017\)On calibration of modern neural networks\.InProceedings of the International Conference on Machine Learning,pp\. 1321–1330\.External Links:[Link](https://proceedings.mlr.press/v70/guo17a.html)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p3.1.11.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.11.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p6.1.4.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p7.1.6.2.1.1)\.
- \[70\]J\. MacQueen\(1967\)Some methods for classification and analysis of multivariate observations\.InProceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability,Vol\.1,pp\. 281–297\.Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.2.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p6.1.2.2.1.1)\.
- \[71\]E\. Perez, F\. Strub, H\. de Vries, V\. Dumoulin, and A\. Courville\(2018\)FiLM: visual reasoning with a general conditioning layer\.InProceedings of the AAAI Conference on Artificial Intelligence,External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/11671)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.7.2.1.1)\.
- \[72\]B\. Zhang and R\. Sennrich\(2019\)Root mean square layer normalization\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/1910.07467)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.7.2.1.1)\.
- \[73\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2106.09685)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.7.2.1.1)\.
- \[74\]J\. Puigcerver, C\. Riquelme, B\. Mustafa, and N\. Houlsby\(2024\)From sparse to soft mixtures of experts\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2308.00951)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.7.2.1.1)\.
- \[75\]S\. Yun, D\. Han, S\. J\. Oh, S\. Chun, J\. Choe, and Y\. Yoo\(2019\)CutMix: regularization strategy to train strong classifiers with localizable features\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 6023–6032\.External Links:[Link](https://arxiv.org/abs/1905.04899)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.8.2.1.1)\.
- \[76\]P\. Foret, A\. Kleiner, H\. Mobahi, and B\. Neyshabur\(2021\)Sharpness\-aware minimization for efficiently improving generalization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=6Tm1mposlrM)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.8.2.1.1)\.
- \[77\]B\. Lakshminarayanan, A\. Pritzel, and C\. Blundell\(2017\)Simple and scalable predictive uncertainty estimation using deep ensembles\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/1612.01474)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p4.1.11.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p7.1.5.2.1.1)\.
- \[78\]Y\. Gorishniy, I\. Rubachev, V\. Khrulkov, and A\. Babenko\(2021\)Revisiting deep learning models for tabular data\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.34\.External Links:[Link](https://proceedings.neurips.cc/paper/2021/hash/9d86d83f925f2149e9edb0ac3b49229c-Abstract.html)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p5.1.2.2.1.1),[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p7.1.2.2.1.1)\.
- \[79\]Y\. Wen, D\. Tran, and J\. Ba\(2020\)BatchEnsemble: an alternative approach to efficient ensemble and lifelong learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Sklf1yrYDr)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p5.1.6.2.1.1)\.
- \[80\]R\. Pascanu, T\. Mikolov, and Y\. Bengio\(2013\)On the difficulty of training recurrent neural networks\.InProceedings of the 30th International Conference on Machine Learning,Vol\.28,pp\. 1310–1318\.External Links:[Link](https://proceedings.mlr.press/v28/pascanu13.html)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p5.1.8.2.1.1)\.
- \[81\]T\. Lin, P\. Goyal, R\. Girshick, K\. He, and P\. Dollár\(2017\)Focal loss for dense object detection\.InProceedings of the IEEE International Conference on Computer Vision,pp\. 2980–2988\.External Links:[Link](https://openaccess.thecvf.com/content_iccv_2017/html/Lin_Focal_Loss_for_ICCV_2017_paper.html)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p5.1.9.2.1.1)\.
- \[82\]B\. Efron\(1979\)Bootstrap methods: another look at the jackknife\.The Annals of Statistics7\(1\),pp\. 1–26\.External Links:[Document](https://dx.doi.org/10.1214/aos/1176344552)Cited by:[§D\.3](https://arxiv.org/html/2609.16309#A4.SS3.p6.1.3.2.1.1)\.
## Appendix AAdditional Results on TabArena
Setup\.TabArena\[[8](https://arxiv.org/html/2609.16309#bib.bib11)\]evaluates each model family under a fixed protocol: one default configuration plus200200random configurations drawn from the family’s search space, each fitted as an 8\-fold bagged ensemble\.*Tuned*\(T\) reports the configuration with the best validation score, and*Tuned\+Ensembled*\(T\+E\) reports the greedy weighted ensemble\[[18](https://arxiv.org/html/2609.16309#bib.bib70)\]built from the same200200configurations\. We plug the agent\-generated search spaces of Section[3\.2](https://arxiv.org/html/2609.16309#S3.SS2)into this protocol unchanged for the five families: LightGBM, MLP†, RealMLP, TabM, and TabICLv2\. We mark the agentic version with a suffix\-A\. For the four families other than MLP†the base numbers are the official TabArena entries; TabArena has no MLP entry, so we additionally run the base MLP†ourselves under the identical protocol\. Since the agentic design changes the tuning space and not the default configuration, MLP†and MLP†\-A share their default run, and only their Tuned and T\+E results differ\. Two differences from the setup of Section[4](https://arxiv.org/html/2609.16309#S4)matter when comparing the numbers\. First, TabArena tunes by random search over the200200configurations, whereas our benchmark obtains the tuned results with Optuna TPE, which is a stronger optimizer\. Second, every TabArena configuration is itself an 8\-fold cross\-validated ensemble: it trains eight models, each on seven of the eight folds, scores the configuration on their out\-of\-fold predictions, and averages the test predictions of all eight models\. Our benchmark trains each configuration once and scores it on the validation subset\. Third, we evaluate on TabArena\-Lite, i\.e\., the first of the seven outer cross\-validation splits\.
Results\.Figure[5](https://arxiv.org/html/2609.16309#A1.F5)shows the official Elo leaderboard over the 51 TabArena\-Lite datasets with the agentic variants included\. The picture from Section[4\.1](https://arxiv.org/html/2609.16309#S4.SS1)transfers: at the T\+E level, the agentic space improves the Elo of MLP†\(from 1295 to 1421\), TabM \(from 1414 to 1497\), RealMLP \(from 1502 to 1557\), and TabICLv2 \(from 1573 to 1612\), and roughly matches it for LightGBM \(from 1402 to 1409\)\. The ordering across families also matches Section[4\.1](https://arxiv.org/html/2609.16309#S4.SS1): the plain MLP†, whose base space is the least tuned, gains the most \(\+125\+125\), while the meta\-tuned RealMLP and the saturated LightGBM gain the least\. Notably, both TabICLv2\-A and RealMLP\-A at the T\+E level outperform thebest\_qualitypreset of AutoGluon 1\.4 \(1535\)111Ensemble of LightGBM, CatBoost, XGBoost, random forests, and MLP\., and TabICLv2\-A is behind only theextreme\_qualitypreset of AutoGluon 1\.5 \(1669\)222Adds TabPFNv2, TabICL, Mitra, TabM, RealTabPFN\-2/2\.5, TabDPT, TabPrep\-LightGBM, and EBM to the 1\.4 ensemble\.and newer TFMs\. At the tuned\-only level the gains are smaller \(\+79\+79for MLP†,\+14\+14for RealMLP,\+3\+3for TabM\) or negative \(−53\-53for LightGBM\): random search is less sample\-efficient in the larger agentic spaces, so, as in Section[4\.1](https://arxiv.org/html/2609.16309#S4.SS1), most of the benefit is realized when the diversity of the pool is exploited by ensembling\. We note that Elo is fitted over the whole method pool, so all ratings shift slightly whenever the pool changes; the numbers above are therefore comparable within this leaderboard but not against other versions of it\. Per\-dataset Default/Tuned/T\+E scores for all five families are reported in Appendix[F](https://arxiv.org/html/2609.16309#A6)\.
Figure 5:Official TabArena\-Lite Elo leaderboard \(51 datasets\) with the agent\-augmented spaces included as LightGBM\-A, MLP\-A, RealMLP\-A, TabM\-A, and TabICLv2\-A\. For each method the bars show the Elo of the default configuration, the best of200200random configurations \(Tuned\), and the greedy ensemble built from the same configurations \(Tuned\+Ensembled\); whiskers are95%95\\%confidence intervals, dashed lines mark the AutoGluon reference systems, and hatched bars are partially imputed methods\.
## Appendix BSetup
### B\.1Hardware
Most of the experiments were conducted on a single NVIDIA A100 GPU paired with AMD EPYC CPU\. Absolute wall\-clock time depends on hardware, parallelism, and cluster load; the agent generation times of Table[6](https://arxiv.org/html/2609.16309#S4.T6)additionally depend on provider load and network latency, so they should be read as indicative rather than reproducible\.
### B\.2Datasets
We provide a full list of datasets and their properties in Table[9](https://arxiv.org/html/2609.16309#A2.T9), which also includes the evaluation metric and feature counts\. The evaluation metric of each dataset is the native metric adopted from the prior work that introduced the corresponding benchmark \(TabM\[[2](https://arxiv.org/html/2609.16309#bib.bib3)\], TabArena\[[8](https://arxiv.org/html/2609.16309#bib.bib11)\], and TabReD\[[7](https://arxiv.org/html/2609.16309#bib.bib12)\]\)\. The main text uses the aggregate summary in Table[1](https://arxiv.org/html/2609.16309#S3.T1)\. We categorize datasets by size as follows: the*large*group consists of the TabReD benchmark datasets together with Microsoft; the remaining datasets are split into*small*and*medium*at the median number of cells \(rows×\\timesfeatures\), with the split boundary at3\.0×1053\.0\\times 10^\{5\}cells\.
Table 9:Per\-dataset statistics\. Size is the small/medium/large bucket used in the paper comparisons \(large = TabReD datasets and Microsoft; the rest split at the median cell count\)\. Metric is the dataset\-native evaluation metric adopted from the source benchmark\. \#Samples is the total number of rows\. For regression we report \#Classes as 0\.DatasetTaskSizeMetric\#Num\.\#Cat\.\#Bin\.\#Classes\#SamplesSourceairfoil\_self\_noiseregsmallrmse41001,503OpenMLAnother\-Dataset\-on\-used\-Fiat\-500regsmallrmse61001,538OpenMLblack\-fridayregmediumrmse5400166,821OpenMLcaliforniaregsmallrmse800020,640OpenMLconcrete\_compressive\_strengthregsmallrmse80001,030OpenMLdiamondregmediumrmse630053,940OpenMLFood\_Delivery\_Timeregmediumrmse630045,451OpenMLhealthcare\_insurance\_expensesregsmallrmse33001,338OpenMLhouseregmediumrmse1600022,784OpenMLmiami\_housingregsmallrmse1410013,776OpenMLmicrosoftreglargermse1360001,200,192OpenMLphysiochemical\_proteinregmediumrmse900045,730OpenMLQSAR\-TID\-11regmediumrmse10240005,742OpenMLQSAR\_fish\_toxicityregsmallrmse6000907OpenMLwine\_qualityregsmallrmse111006,497OpenMLcooking\-timereglargermse186330319,986TabReDdelivery\-etareglargermse221110416,451TabReDmaps\-routingreglargermse984200340,981TabReDsberbank\-housingreglargermse3651017028,321TabReDweatherreglargermse100030423,795TabReDAmazon\_employee\_accessclfsmallroc\_auc090232,769OpenMLAPSFailureclfmediumroc\_auc17000276,000OpenMLbank\-marketingclfmediumroc\_auc580245,211OpenMLBioresponseclfmediumroc\_auc17760023,751OpenMLchurnclfsmallaccuracy1010210,000OpenMLcredit\_card\_clients\_defaultclfmediumroc\_auc2030230,000OpenMLcustomer\_satisfaction\_in\_airlineclfmediumroc\_auc51602129,880OpenMLdiabetesclfsmallroc\_auc8002768OpenMLDiabetes130USclfmediumroc\_auc8390271,518OpenMLE\-CommereShippingDataclfsmallroc\_auc640210,999OpenMLGiveMeSomeCreditclfmediumroc\_auc10002150,000OpenMLhelocclfsmallroc\_auc2300210,459OpenMLHR\_Analytics\_Job\_Change\_of\_Data\_Scientistsclfsmallroc\_auc2100219,158OpenMLin\_vehicle\_coupon\_recommendationclfmediumroc\_auc2220212,684OpenMLjm1clfsmallroc\_auc2100210,885OpenMLkddcup09\_appetencyclfmediumroc\_auc174380250,000OpenMLNATICUSdroidclfmediumroc\_auc086027,491OpenMLonline\_shoppers\_intentionclfsmallroc\_auc1061212,330OpenMLpolish\_companies\_bankruptcyclfmediumroc\_auc640025,910OpenMLqsar\-biodegclfsmallroc\_auc365021,054OpenMLspliceclfsmalllog\_loss060033,190OpenMLtaiwanese\_bankruptcy\_predictionclfmediumroc\_auc940026,819OpenMLecom\-offersclflargeroc\_auc113062160,057TabReDhomecredit\-defaultclflargeroc\_auc6128222381,664TabReDhomesite\-insuranceclflargeroc\_auc25323232260,753TabReD
### B\.3Metrics
Each dataset is evaluated with its native metric \(Appendix[B\.2](https://arxiv.org/html/2609.16309#A2.SS2)\)\. To compare methods within a dataset, we use the signed scoresB\(m,𝒟\)s\_\{B\}\(m;\\mathcal\{D\}\): the metric value at budgetBBfor lower\-is\-better metrics \(RMSE, log\-loss\) and its negation for higher\-is\-better ones \(AUROC, accuracy\), so that lowersBs\_\{B\}is always better\. We report four aggregate metrics; as discussed in[Erickson et al\. \[8\]](https://arxiv.org/html/2609.16309#bib.bib11), they offer complementary views on the performance\.
- •Mean rank\.Pool every \(model family, search space\) pair on a dataset, sort bysBs\_\{B\}, and letrank\(m,𝒮,𝒟\)\\mathrm\{rank\}\(m,\\mathcal\{S\};\\mathcal\{D\}\)be the position of\(m,𝒮\)\(m,\\mathcal\{S\}\)in the sorted ordering \(rank11= best\)\. The mean rank averages this across datasets\.
- •Relative improvement\.Effect of switching to methodmmfrom a reference methodrr\(the tuned base\-space MLP†, unless stated otherwise; Table[4](https://arxiv.org/html/2609.16309#S4.T4)uses the model’s own base space asrr\): ΔB\(m,𝒟\)=100sB\(r,𝒟\)−sB\(m,𝒟\)\|sB\(r,𝒟\)\|,\\Delta\_\{B\}\(m;\\mathcal\{D\}\)=100\\,\\frac\{s\_\{B\}\(r;\\mathcal\{D\}\)\-s\_\{B\}\(m;\\mathcal\{D\}\)\}\{\\left\|s\_\{B\}\(r;\\mathcal\{D\}\)\\right\|\},\(1\)averaged across datasets\. BecauseΔB\\Delta\_\{B\}is defined on the signed native metric, its denominator is the metric magnitude itself \(e\.g\.\|AUROC\|\|\\mathrm\{AUROC\}\|rather than\|1−AUROC\|\|1\-\\mathrm\{AUROC\}\|, which would inflate small absolute changes on near\-saturated datasets\)\.
- •Normalized score\.The only metric computed on the error scale:err=1−Score\\mathrm\{err\}=1\-\\mathrm\{Score\}for the two higher\-is\-better metrics in our suite, i\.e\.err=1−AUROC\\mathrm\{err\}=1\-\\mathrm\{AUROC\}anderr=1−accuracy\\mathrm\{err\}=1\-\\mathrm\{accuracy\}, while the lower\-is\-better RMSE and log\-loss are used as is,err=Score\\mathrm\{err\}=\\mathrm\{Score\}\. Per dataset, errors are linearly rescaled so that the best method maps to11, the median method maps to00, and anything worse than the median is truncated to00; higher is better\[[50](https://arxiv.org/html/2609.16309#bib.bib74)\]\.
- •Elo rating\.Each per\-dataset comparison between two methods is a logistic “match” \(the method with the better signed score wins\); we fit ratings by minimizing the Bradley–Terry log\-loss with L\-BFGS over all pairs and dataset/budget combinations following[Chiang et al\. \[51\]](https://arxiv.org/html/2609.16309#bib.bib73), then shift the resulting scale so that the MLP†baseline anchors at10001000\. Larger Elo means stronger overall performance\.
Together, rank and Elo capture the relative standing, while relative improvement and normalized score summarize the scale of the change\.
### B\.4Hyperparameters
We use model\-specific HPO spaces that match strong published tabular baselines, then compare each baseline against its agent\-augmented counterpart under the same split, metric, sampler, and trial budget\.
#### MLP†\.
Our MLP baseline follows the feed\-forward baseline used with TabM: a stack of fully connected blocks with tuned depth, width, dropout, learning rate, weight decay, and piecewise\-linear numerical embeddings\[[2](https://arxiv.org/html/2609.16309#bib.bib3),[32](https://arxiv.org/html/2609.16309#bib.bib9)\]\. The PLR embedding bins and embedding dimension are part of the HPO space because numerical embeddings are a major driver of neural tabular performance\. The full tuning space is shown in Table[10](https://arxiv.org/html/2609.16309#A2.T10)\.
Table 10:The hyperparameter tuning space for MLP†\.
#### TabM†\.
TabM is a parameter\-efficient neural ensemble that shares most parameters across ensemble members while producingkkpredictions per example\[[2](https://arxiv.org/html/2609.16309#bib.bib3)\]\. We use the mini TabM configuration withk=32k=32, tune the backbone width, depth, dropout, PLR embedding parameters, learning rate, and weight decay, and keep the same early\-stopping and gradient\-clipping protocol as for MLP\. The full tuning space is shown in Table[11](https://arxiv.org/html/2609.16309#A2.T11)\.
Table 11:The hyperparameter tuning space for TabM†\.
#### RealMLP\.
RealMLP is included as a strong pre\-tuned modular MLP baseline designed to be competitive without extensive manual retuning\[[1](https://arxiv.org/html/2609.16309#bib.bib15)\]\. We tune the RealMLP choices exposed in its recommended configuration family, including front scaling, dropout, activation, hidden\-size template, numerical embedding type, PLR scale, learning rate, weight decay, and label smoothing for classification\. The full tuning space is shown in Table[12](https://arxiv.org/html/2609.16309#A2.T12)\.
Table 12:The hyperparameter tuning space for RealMLP\.
#### LightGBM\.
LightGBM is a histogram\-based gradient\-boosted decision tree method\[[33](https://arxiv.org/html/2609.16309#bib.bib33)\]\. We use the HPO ranges adopted by the TabR benchmark protocol\[[15](https://arxiv.org/html/2609.16309#bib.bib8)\], tuning feature and bagging fractions, learning rate, leaf count, L2 regularization, and the minimum Hessian mass per leaf\. The number of estimators is fixed at40004000with early stopping after200200rounds\. The full tuning space is shown in Table[13](https://arxiv.org/html/2609.16309#A2.T13)\.
Table 13:The hyperparameter tuning space for LightGBM\.
#### TabICLv2\.
TabICLv2 is a tabular foundation model for in\-context learning on large tabular datasets\[[16](https://arxiv.org/html/2609.16309#bib.bib38)\]\. We keep the publishedTabICLv2checkpoint fixed and do not train or fine\-tune the backbone\. We fit the model by passing the concatenated train and validation portions of the dataset as the context: since TabICLv2 performs no gradient\-based training, the validation split is not needed for early stopping and would otherwise go unused, whereas the other families consume it for early stopping and model selection\. To avoid out\-of\-memory issues, we subsample the context to the minimum of40,00040,000samples or3,840,000/nfeatures\{3,840,000\}/\{n\_\{\\text\{features\}\}\}\.
### B\.5Hyperparameter optimization and ensembling
#### Hyperparameter optimization\.
We use the univariate TPE sampler in Optuna to tune individual model hyperparameters\[[38](https://arxiv.org/html/2609.16309#bib.bib16),[39](https://arxiv.org/html/2609.16309#bib.bib32)\]\. For small and medium datasets, we set the budget to 200 trials; for large ones \(TabReD and Microsoft\), to 100 trials\. The exception is TabICLv2: the default model is evaluated as published, without any tuning, and the agentic TabICLv2 is tuned with a fixed budget of 100 trials on all datasets\. In both cases, the first 20 trials use the random sampler to encourage exploration of the space\. Empirically, we find the default univariate TPE sampler with the above budgets to work well, and we leave the search for better defaults to future work\. For example, the number of random startup trials could be chosen adaptively based on the Coupon Collector’s problem\[[52](https://arxiv.org/html/2609.16309#bib.bib78)\]\.
#### Ensembles\.
We build greedy ensembles following[Caruana et al\. \[18\]](https://arxiv.org/html/2609.16309#bib.bib70)\. We first sample 100 random configurations \(20 for large datasets\) from the given HPO space and train a model for each\. We then iteratively grow the ensemble for a fixed budget of steps: at each step, we try adding each fitted model \(selection is with replacement\), average its predictions with those of the already selected models, and keep the candidate that yields the lowest validation error\. Note that we do not use early stopping: the ensemble is grown for the full budget even if the validation error stops improving at some intermediate step\.
## Appendix CFuture Research Directions
Beyond the aggregate gains, the per\-axis selection patterns in Figure[3](https://arxiv.org/html/2609.16309#S5.F3)reveal a structural pattern in how tabular research progresses\. On three of the four most important modules \(the model, the training procedure, and the numerical preprocessing\), the non\-default agent\-proposed implementations are selected most often\. The fourth module, numerical embeddings, is the only one whose default remains competitive, and it is also the one that has received by far the most attention in the recent tabular deep learning\[[32](https://arxiv.org/html/2609.16309#bib.bib9),[2](https://arxiv.org/html/2609.16309#bib.bib3),[1](https://arxiv.org/html/2609.16309#bib.bib15)\]\. The other three modules, by comparison, remain underexplored: while isolated efforts exist – for instance, recent work on optimizers for tabular deep learning\[[53](https://arxiv.org/html/2609.16309#bib.bib72)\]– much else is overlooked\. Therefore, we believe more investigation into the other pipeline modules is needed\.
We attribute this pattern to two complementary factors\. First, there is a long tail of plausible\-but\-forgotten methods on each underexplored axis, displaced by inertia rather than by evidence\. Second, the breadth of the hypothesis pool available to an agent far exceeds that of a human researcher: an agent can inexpensively “swarm” the literature for plausible candidates whenever the user requests it, with no commitment to championing any single one\. On the other hand, researchers and machine learning practitioners tend to concentrate on a few well\-established options\. The combination – a long tail of forgotten methods and a wide retrieval window – is, in our view, the principal reason agent\-augmented HPO improves over fixed search spaces precisely on the underexplored axes\. We view a systematic study of such overlooked axes – now cheap to enumerate via agentic generation – as a productive direction in itself\.
## Appendix DAgent Setup and Artifacts
Between the runs and before the first use, we sanitize the agent environment and memory to avoid potential leaks or biases by running the following procedure:
- •Remove memory and session information via - –rm \-rf /\.claude/projects/<\.\.\.\>/memory - –rm \-rf /\.claude/projects/<\.\.\.\>/sessions
- •Checkout a clean branch:git checkout \-b <semiagent\-version\-x\>
- •Sanitizegithistory - –git reflog expire \-\-expire=now \-\-all - –git gc \-\-prune=now \-\-quiet
### D\.1AutoML baselines
#### Protocol\.
We run AutoKeras 1\.1\.0 \(TensorFlow 2\.15\), AutoPyTorch 0\.2\.1, and AutoGluon 1\.5\.0 unmodified on the 45 datasets with our exact test splits\. We pass our validation split directly into the system: AutoPyTorch via a custom holdout, AutoKeras viavalidation\_data\)\. AutoGluon performs its own internal bagging, so we pass it concatenated train and validation subsets\. We set the budget to 4 hours for each system following[Erickson et al\. \[8\]](https://arxiv.org/html/2609.16309#bib.bib11),[Gijsbers et al\. \[47\]](https://arxiv.org/html/2609.16309#bib.bib82)\. AutoPyTorch returns both the best single model and the greedy ensemble\. AutoKeras returns the best model\. AutoGluon returns a single predictor, which is a bagged and stacked ensemble\.
#### Library size\.
Table[14](https://arxiv.org/html/2609.16309#A4.T14)maps the search spaces of the two neural AutoML systems onto our module axes and counts interchangeable implementations, excluding the default on each axis\. AutoPyTorch’s2222numerical\-preprocessing components are77scalers and the variance threshold plus1515feature\-preprocessing transforms it inherits from auto\-sklearn\[[17](https://arxiv.org/html/2609.16309#bib.bib14)\]; its88architecture components are44backbones and44weight initializers, and its77training components are66learning\-rate schedulers and MixUp\. AutoKeras exposes normalization and batch normalization as on/off switches and two optimizer alternatives; the rest of its structured\-data space is numeric \(depth, width, dropout, learning rate\)\.
Table 14:Interchangeable components of the two neural AutoML libraries and of the agentic MLP†space, mapped onto our module axes; counts exclude the default implementation on each axis\. Axes dispatched by task are reported as regression/classification\.
#### AutoGluon portfolio\.
Underbest\_quality, AutoGluon fits a fixed, meta\-learned zero\-shot portfolio of 110 configurations \(Table[15](https://arxiv.org/html/2609.16309#A4.T15)\), bags each configuration with 8\-fold cross\-validation, feeds the out\-of\-fold predictions to a second stack layer of the same model families, and combines the resulting models with greedy weighted ensembling\. In theextreme\_qualitypreset, AutoGluon replaces the portfolio of 110 with a portfolio of 28 configurations which also includes TabM and tabular foundation models \(RealTabPFN\-v2, TabDPT, TabICL, Mitra\)\.
Table 15:Composition of AutoGluon’s zero\-shot portfolios:best\_quality\(v1\.5\.0, as resolved in our runs\) andextreme\_quality\(v1\.5, portfoliozeroshot\_2025\_12\_18\_gpu\)\. Entries are the number of configurations per model family\.
#### Agentic hand\-off \(AutoGluon \(\+A\)\)\.
We add 8 randomly\-sampled configurations of the agentic RealMLP and TabM†models\. The additional agentic configurations are interleaved with the stock portfolio and are bagged, stacked, and ensembled exactly like the stock members under the same four\-hour budget \(126126configurations in total\)\.
### D\.2Hypothesis\-generation workflow
```
bin/modules/
+-- embedding/
| +-- __init__.py
| +-- one_hot.py
+-- loss_reg/
+-- __init__.py
+-- huber.py
```
Figure 6:Module registry package structure underbin/modules\. Each subdirectory corresponds to one pipeline stage; individual\.pyfiles implement registered module choices for that axis\.#### Agent environment\.
Each model family is handled by a single agent run\. The agent is given the repository for that family: a small pipeline in which each module axis \(preprocessing, embedding, model, training, optimizer, loss, and inference\) is represented by a package whose registry maps integer indices to builder functions \(Figure[6](https://arxiv.org/html/2609.16309#A4.F6)\)\. The agent is equipped with two main tools:inspect\_pipeline\.py, which prints the registered modules across all axes, andtest\.py, which runs a synthetic\-data smoke test for a chosen module combination\.
#### Workflow\.
After a preflight check that the environment is set up and the baseline runs, the agent proceeds in four stages:*inspection*,*hypothesis generation*,*implementation*, and*testing*\. During*inspection*, the agent maps out the repository structure and the model pipeline\. During*hypothesis generation*, it proposes up to\-\-num\-hypothesescandidate ideas, spread across the axes, and filters out those that do not meet our evidence bar: an idea must \(a\) have a clear mechanism by which it affects the model, \(b\) have support in the literature, \(c\) not replicate something already present in the base model, and \(d\) be benign by default, i\.e\., not harm performance when inactive\. The surviving hypotheses are then implemented one at a time in an implement\-and\-test loop\. For each module, the agent: \(1\) implements it as a new self\-contained\.pyfile; \(2\) registers it in the package’s\_\_init\_\_\.py; \(3\) runs the smoke test \(uv run test\.py \-\-<axis\>\-idx <idx\>\); \(4\) reads any performance and efficiency warnings; and \(5\) debugs the module, or drops it if it degrades runtime or memory by more than 10x times\. After all hypotheses are processed, the agent tests module*interactions*by runninguv run test\.py \-\-exhaustiveand debugs or discards any problematic combinations\. The full candidate sets are provided in Tables[D\.3](https://arxiv.org/html/2609.16309#A4.SS3)–[D\.3](https://arxiv.org/html/2609.16309#A4.SS3)\.
#### Synthetic dataset test\.
Thetest\.pyscript evaluates each candidate against the all\-v0baseline on 4 temporary synthetic datasets generated inside the test runner\. Each scenario hasN=Ntrain\+Nval\+NtestN=N\_\{\\mathrm\{train\}\}\+N\_\{\\mathrm\{val\}\}\+N\_\{\\mathrm\{test\}\}rows,ppnumerical features, andqqcategorical features, with\(p,q\)∈\{\(100,100\),\(12,12\),\(16,0\),\(0,16\)\}\(p,q\)\\in\\\{\(100,100\),\(12,12\),\(16,0\),\(0,16\)\\\}\. For numerical features, the runner samples
xijnum=zij\+tiaj,zij∼𝒩\(0,1\),x^\{\\mathrm\{num\}\}\_\{ij\}=z\_\{ij\}\+t\_\{i\}a\_\{j\},\\qquad z\_\{ij\}\\sim\\mathcal\{N\}\(0,1\),wheretit\_\{i\}is linearly spaced from−1\-1to11over rows andaja\_\{j\}is linearly spaced from0\.050\.05to0\.50\.5over numerical columns\. Categorical features are sampled as integer codes\. By default,
xijcat∼Unif\{0,…,5\}\.x^\{\\mathrm\{cat\}\}\_\{ij\}\\sim\\mathrm\{Unif\}\\\{0,\\ldots,5\\\}\.In the\(12,12\)\(12,12\)categorical edge\-case scenario, the first four categorical columns use
xijcat∼Unif\{0,…,3\}on train,xijcat∼Unif\{0,…,4\}on validation/test,x^\{\\mathrm\{cat\}\}\_\{ij\}\\sim\\mathrm\{Unif\}\\\{0,\\ldots,3\\\}\\quad\\text\{on train\},\\qquad x^\{\\mathrm\{cat\}\}\_\{ij\}\\sim\\mathrm\{Unif\}\\\{0,\\ldots,4\\\}\\quad\\text\{on validation/test\},with category44forced to appear outside training\. The next four columns use the converse stress case,
xijcat∼Unif\{0,…,6\}on train,xijcat∼Unif\{0,…,2\}on validation/test,x^\{\\mathrm\{cat\}\}\_\{ij\}\\sim\\mathrm\{Unif\}\\\{0,\\ldots,6\\\}\\quad\\text\{on train\},\\qquad x^\{\\mathrm\{cat\}\}\_\{ij\}\\sim\\mathrm\{Unif\}\\\{0,\\ldots,2\\\}\\quad\\text\{on validation/test\},with category66forced to appear in training\. Remaining categorical columns useUnif\{0,…,4\}\\mathrm\{Unif\}\\\{0,\\ldots,4\\\}in all splits\.
Targets are generated from a finite signal plus noise\. Let
si=∑j=1min\(p,8\)bjxijnum\+∑j=1min\(q,8\)cj\(xijcatmod7\)\+ϵi,ϵi∼𝒩\(0,0\.12\),s\_\{i\}=\\sum\_\{j=1\}^\{\\min\(p,8\)\}b\_\{j\}x^\{\\mathrm\{num\}\}\_\{ij\}\+\\sum\_\{j=1\}^\{\\min\(q,8\)\}c\_\{j\}\(x^\{\\mathrm\{cat\}\}\_\{ij\}\\bmod 7\)\+\\epsilon\_\{i\},\\qquad\\epsilon\_\{i\}\\sim\\mathcal\{N\}\(0,0\.1^\{2\}\),wherebjb\_\{j\}is linearly spaced from0\.20\.2to1\.01\.0andcjc\_\{j\}from0\.030\.03to0\.110\.11\. Regression targets areyi=siy\_\{i\}=s\_\{i\}\. Binary labels areyi=𝟏\{si\>mediank∈train\(sk\)\}y\_\{i\}=\\mathbf\{1\}\\\{s\_\{i\}\>\\mathrm\{median\}\_\{k\\in\\mathrm\{train\}\}\(s\_\{k\}\)\\\}\. Multiclass labels are obtained by discretizingsis\_\{i\}at the training\-set tertiles\. The runner uses a short two\-epoch pass by default and a twelve\-epoch long pass to expose modules that activate only after warmup or several epochs, such as schedulers, exponential moving averages\[[40](https://arxiv.org/html/2609.16309#bib.bib60)\], or training\-loop changes\. It checks that predictions and metrics are finite, and reports wall time, iteration speed, trainable parameter count, process memory, and GPU memory when CUDA is used\.
### D\.3Autoresearch details
We adapt the autoresearch repository framework for tabular data and describe its mechanics below\[[30](https://arxiv.org/html/2609.16309#bib.bib2)\]\. Figure[7](https://arxiv.org/html/2609.16309#A4.F7)shows the per\-iteration trajectories on the four evaluated datasets\. In Table[16](https://arxiv.org/html/2609.16309#A4.T16), we compare the autoresearch best proposed solution to the matched trial\-budget solutions obtained by the agentic and classical HPO\. We see that the classical HPO matches or beats autoresearch on three of the four datasets, while agentic HPO further improves on the classical version everywhere\. We note that due to the cost \(varying from2020to5050USD per run\), we do not re\-run the autoresearch, so we report a single run without error bars\.
Figure 7:Autoresearch onMLP†\\mathrm\{MLP\}^\{\\dagger\}: 50 iterations on four datasets\. Validation improves only marginally and trails classical HPO and our agentic HPO; test does not consistently follow validation\.Table 16:Comparison of Autoresearch, Default HPO and Agentic HPO at a budget of 50 trials each on theMLP†\\mathrm\{MLP\}^\{\\dagger\}pipeline\.Unlike our method, which uses the agent only to propose candidate module implementations, the autoresearch baseline of Section[3\.2](https://arxiv.org/html/2609.16309#S3.SS2)lets the agent edit the model end to end\. It runs an autonomous loop on a single regression or classification task\. Starting from the baselinetrain\.py, each step \(i\) editstrain\.py—any hyperparameter, preprocessing, embedding, optimizer, loss, training\-loop, or architecture change is allowed; \(ii\) commits the change; \(iii\) runsbenchmark\.py, which prints a single validation score; and \(iv\) keeps the commit if validation improves and reverts it \(git reset \-\-hard\) otherwise, appending every trial to a results log\. The data loaderprepare\.pyand the entry pointbenchmark\.pyare read\-only, the held\-out test split is never exposed, and the model must remain a single network of the corresponding family \(no model swap, no ensembling of independently trained models, no new dependencies\)\. The loop runs for a fixed iteration count\. Because it edits one dataset at a time and accepts changes greedily on validation, it neither amortizes generation across datasets nor captures the module interactions that joint HPO exploits, which is reflected in its weaker results in Table[16](https://arxiv.org/html/2609.16309#A4.T16)\.
Table 17:Agent\-proposed module candidates for MLP†\(67 candidates, deduplicated across the regression and classification branches\)\. Counts exclude the defaultv0module on each axis\.Table 18:Agent\-proposed module candidates for TabM†\[[2](https://arxiv.org/html/2609.16309#bib.bib3)\]\(56 candidates\)\. Counts exclude the defaultv0module on each axis\.Table 19:Agent\-proposed module candidates for RealMLP\[[1](https://arxiv.org/html/2609.16309#bib.bib15)\]\(13 candidates\)\. Counts exclude the defaultv0module on each axis\.Table 20:Agent\-proposed module candidates for LightGBM\[[33](https://arxiv.org/html/2609.16309#bib.bib33)\]\(48 candidates\)\. The base learner is fixed; candidates act before \(feature, augmentation\) or after \(post\-processing\) the learner\. Counts exclude the defaultv0module on each axis\.Table 21:Agent\-proposed module candidates and implementation baskets for TabICLv2\[[5](https://arxiv.org/html/2609.16309#bib.bib5),[16](https://arxiv.org/html/2609.16309#bib.bib38)\]\(standard module pool: 14\)\. The foundation\-model backbone is frozen, so candidates act on inputs \(preprocessing, permutation, augmentation\) or outputs \(ensembling, post\-processing\)\. Basket rows indicate module baskets used for ensembles\.
## Appendix EExtended Results
This appendix reports extended results\. The default variant searches only the model\-specific space𝒮HPO\\mathcal\{S\}\_\{\\mathrm\{HPO\}\}described in Appendix[B](https://arxiv.org/html/2609.16309#A2); the agentic variant searches the augmented space𝒮agent\\mathcal\{S\}\_\{\\mathrm\{agent\}\}, which contains the default module choices as a subset\. We provide the full list of agent\-generated hypotheses in Tables[D\.3](https://arxiv.org/html/2609.16309#A4.SS3)–[D\.3](https://arxiv.org/html/2609.16309#A4.SS3)\.
For each model family, the agent\-generated candidate set is produced once and then reused across all datasets\. The agent does not receive dataset contents or dataset\-specific feedback during generation\. Downstream selection is performed only by HPO: each trial is trained on the training split, selected by validation performance, and evaluated on the held\-out test split after selection\. We report the mean and standard deviation of the test metric over repeated evaluation seeds\.
### E\.1Ablation study
Table 22:Per\-run results behind Table[7](https://arxiv.org/html/2609.16309#S5.T7): each row is one agent\-generated MLP†search space evaluated on all 45 datasets in the joint 12\-space pool\. Win rate is the fraction of datasets on which the run improves over the tuned base\-space MLP†\.Table[22](https://arxiv.org/html/2609.16309#A5.T22)reports the per\-run agent results behind the summary in Section[5\.2](https://arxiv.org/html/2609.16309#S5.SS2)\. All runs use the agentic MLP†pipeline under the same split, metric, validation rule, and trial budget as the main experiments\.
### E\.2Full per\-dataset results
We report full per\-dataset results extracted at the terminal budget: 200 trials for small and medium datasets and 100 trials for the large ones in Table[E\.2](https://arxiv.org/html/2609.16309#A5.SS2)\. Below the family rows, each subtable also lists the individual\-model rows of AutoPyTorch and AutoKeras; these are single unseeded runs, so no standard deviation is reported\.
Table 23:Individual results for each of the 45 datasets\. Each subtable reports the terminal\-budget mean test metric±\\pmstandard deviation for Default HPO and Agentic HPO under the same split, metric, and HPO budget\. Arrows indicate whether higher or lower values are better\.black\-friday \(×103\\times 10^\{3\}\)↓\\downarrow
california \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
churn \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
diamond \(×102\\times 10^\{2\}\)↓\\downarrow
house \(×104\\times 10^\{4\}\)↓\\downarrow
microsoft \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
tabarena/airfoil\_self\_noise↓\\downarrow
tabarena/Amazon\_employee\_access \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/Another\-Dataset\-on\-used\-Fiat\-500 \(×102\\times 10^\{2\}\)↓\\downarrow
tabarena/APSFailure \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/bank\-marketing \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/Bioresponse \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/concrete\_compressive\_strength↓\\downarrow
tabarena/credit\_card\_clients\_default \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/customer\_satisfaction\_in\_airline \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/diabetes \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/Diabetes130US \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/E\-CommereShippingData \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/Food\_Delivery\_Time↓\\downarrow
tabarena/GiveMeSomeCredit \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/healthcare\_insurance\_expenses \(×103\\times 10^\{3\}\)↓\\downarrow
tabarena/heloc \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/HR\_Analytics\_Job\_Change\_of\_Data\_Scientists \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/in\_vehicle\_coupon\_recommendation \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/jm1 \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/kddcup09\_appetency \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/miami\_housing \(×104\\times 10^\{4\}\)↓\\downarrow
tabarena/NATICUSdroid \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/online\_shoppers\_intention \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/physiochemical\_protein↓\\downarrow
tabarena/polish\_companies\_bankruptcy \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/qsar\-biodeg \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/QSAR\-TID\-11 \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
tabarena/QSAR\_fish\_toxicity \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
tabarena/splice \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
tabarena/taiwanese\_bankruptcy\_prediction \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/wine\_quality \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
tabred/cooking\-time \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
tabred/delivery\-eta \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
tabred/ecom\-offers \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabred/homecredit\-default \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabred/homesite\-insurance \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabred/maps\-routing \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
tabred/sberbank\-housing \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
tabred/weather↓\\downarrow
### E\.3Ensemble results
Table[E\.3](https://arxiv.org/html/2609.16309#A5.SS3)reports the per\-dataset greedy\-ensemble results behind the ensembling discussion in Section[4\.1](https://arxiv.org/html/2609.16309#S4.SS1)\. Each subtable lists the four families \(MLP†, TabM†, RealMLP, and LightGBM\) with their base and agentic search spaces, and for each \(family, space\) reports three quantities: the ensemble’s test metric \(*Ens\.*\), the test metric of the single strongest ensemble member \(*Best*\), and the mean pairwise test\-prediction correlation across the selected members \(ρ\\rho; lower→\\rightarrowmore diverse pool\)\. Ensembles are constructed by greedy forward selection following[Caruana et al\. \[18\]](https://arxiv.org/html/2609.16309#bib.bib70)on a pool of100100\(2020for the large datasets\) random\-hyperparameter configurations per \(family, space\)\. In the few cases where greedy selection collapses to a single distinct member, the pairwise correlation is undefined, which we denote by “–” in the table\. Below the family rows, each subtable lists the ensemble rows of the AutoPytorch and AutoGluon \(see Appendix[D\.1](https://arxiv.org/html/2609.16309#A4.SS1)for details\)\. We include the default AutoGluon portfolio under*Default*column, and AutoGluon with agentic RealMLP and TabM†under*Agentic*\. Arrows indicate whether higher or lower values are better\.
Table 24:Per\-dataset greedy\-ensemble results for each of the 45 datasets\. Each subtable reports, for the default and agentic search spaces of every model family, the ensemble test metric \(Ens\.\), the test metric of its strongest single member \(Best\), and the mean pairwise prediction correlation among selected members \(ρ\\rho\)\. Arrows indicate whether higher or lower values are better\.black\-friday \(×103\\times 10^\{3\}\)↓\\downarrow
california \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
churn \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
diamond \(×102\\times 10^\{2\}\)↓\\downarrow
house \(×104\\times 10^\{4\}\)↓\\downarrow
microsoft \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
tabarena/airfoil\_self\_noise↓\\downarrow
tabarena/Amazon\_employee\_access \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/Another\-Dataset\-on\-used\-Fiat\-500 \(×102\\times 10^\{2\}\)↓\\downarrow
tabarena/APSFailure \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/bank\-marketing \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/Bioresponse \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/concrete\_compressive\_strength↓\\downarrow
tabarena/credit\_card\_clients\_default \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/customer\_satisfaction\_in\_airline \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/diabetes \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/Diabetes130US \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/E\-CommereShippingData \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/Food\_Delivery\_Time↓\\downarrow
tabarena/GiveMeSomeCredit \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/healthcare\_insurance\_expenses \(×103\\times 10^\{3\}\)↓\\downarrow
tabarena/heloc \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/HR\_Analytics\_Job\_Change\_of\_Data\_Scientists \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/in\_vehicle\_coupon\_recommendation \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/jm1 \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/kddcup09\_appetency \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/miami\_housing \(×104\\times 10^\{4\}\)↓\\downarrow
tabarena/NATICUSdroid \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/online\_shoppers\_intention \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/physiochemical\_protein↓\\downarrow
tabarena/polish\_companies\_bankruptcy \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/qsar\-biodeg \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/QSAR\-TID\-11 \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
tabarena/QSAR\_fish\_toxicity↓\\downarrow
tabarena/splice \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
tabarena/taiwanese\_bankruptcy\_prediction \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabarena/wine\_quality \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
tabred/cooking\-time \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
tabred/delivery\-eta \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
tabred/ecom\-offers \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabred/homecredit\-default \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabred/homesite\-insurance \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
tabred/maps\-routing \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
tabred/sberbank\-housing \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
tabred/weather↓\\downarrow
## Appendix FTabArena Per\-Dataset Results
Table[F](https://arxiv.org/html/2609.16309#A6)reports the per\-dataset results behind the TabArena\-Lite comparison in Appendix[A](https://arxiv.org/html/2609.16309#A1)\. Each subtable lists the five model families and reports, for the base and the agentic \(\-A\) search space, the score of the default configuration \(Def\.\), of the best of the200200random configurations by validation score \(Tuned\), and of the greedy ensemble built from the same configurations \(T\+E\)\. Scores use the official TabArena metric of each dataset: AUROC for binary classification, log\-loss for multiclass, and RMSE for regression\. Base rows are the official TabArena results, except MLP†, which has no official TabArena counterpart and whose base row is our own run under the same protocol, sharing its default configuration with MLP†\-A\. The official TabICLv2 entry is default\-only, so its Tuned and T\+E cells are marked as missing\.
Table 25:Per\-dataset TabArena\-Lite results for the 51 datasets: default \(Def\.\), tuned \(Tuned\), and tuned\-plus\-ensembled \(T\+E\) test scores for the base and agentic search space of each model family\. Arrows indicate whether higher or lower values are better\.airfoil\_self\_noise, RMSE↓\\downarrow
Amazon\_employee\_access, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
anneal, Log\-loss \(×10−2\\times 10^\{\-2\}\)↓\\downarrow
Another\-Dataset\-on\-used\-Fiat\-500, RMSE \(×102\\times 10^\{2\}\)↓\\downarrow
APSFailure, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
bank\-marketing, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
Bank\_Customer\_Churn, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
Bioresponse, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
blood\-transfusion\-service\-center, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
churn, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
coil2000\_insurance\_policies, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
concrete\_compressive\_strength, RMSE↓\\downarrow
credit\-g, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
credit\_card\_clients\_default, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
customer\_satisfaction\_in\_airline, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
diabetes, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
Diabetes130US, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
diamonds, RMSE \(×102\\times 10^\{2\}\)↓\\downarrow
E\-CommereShippingData, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
Fitness\_Club, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
Food\_Delivery\_Time, RMSE↓\\downarrow
GiveMeSomeCredit, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
hazelnut\-spread\-contaminant\-detection, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
healthcare\_insurance\_expenses, RMSE \(×103\\times 10^\{3\}\)↓\\downarrow
heloc, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
hiva\_agnostic, Log\-loss \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
houses, RMSE \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
HR\_Analytics\_Job\_Change\_of\_Data\_Scientists, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
in\_vehicle\_coupon\_recommendation, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
Is\-this\-a\-good\-customer, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
jm1, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
kddcup09\_appetency, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
Marketing\_Campaign, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
maternal\_health\_risk, Log\-loss \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
miami\_housing, RMSE \(×104\\times 10^\{4\}\)↓\\downarrow
MIC, Log\-loss \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
NATICUSdroid, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
online\_shoppers\_intention, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
physiochemical\_protein, RMSE↓\\downarrow
polish\_companies\_bankruptcy, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
qsar\-biodeg, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
QSAR\-TID\-11, RMSE \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
QSAR\_fish\_toxicity, RMSE \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
SDSS17, Log\-loss \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
seismic\-bumps, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
splice, Log\-loss \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
students\_dropout\_and\_academic\_success, Log\-loss \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
superconductivity, RMSE↓\\downarrow
taiwanese\_bankruptcy\_prediction, AUROC \(×10−1\\times 10^\{\-1\}\)↑\\uparrow
website\_phishing, Log\-loss \(×10−1\\times 10^\{\-1\}\)↓\\downarrow
wine\_quality, RMSE \(×10−1\\times 10^\{\-1\}\)↓\\downarrowSimilar Articles
Agentic search models (5 minute read)
Agentic search models are LLMs trained specifically for orchestrating search tasks, offering smaller, faster, and domain-specific alternatives to general models like GPT-5. They unbundle the traditional monolithic search stack by allowing an intelligent model to manage the entire retrieval process.
SearchSwarm: Towards Delegation Intelligence in Agentic LLMs for Long-Horizon Deep Research
This paper introduces SearchSwarm, a model trained on synthesized delegation intelligence to improve long-horizon deep research tasks via task decomposition and subagent coordination, achieving state-of-the-art results on BrowseComp benchmarks.
Exploit More, Explore Smarter for Budget-Constrained Agentic Search
This paper introduces ExTS, a tree-search policy for budget-constrained agentic search in LLM agents that improves over standard baselines in tasks like prompt optimization and code generation with an average +5.5% gain.
How Do Tool-Augmented LLM Agents Perform on Real-World Energy Analytics Tasks?
This paper presents an empirical study and benchmark for evaluating tool-augmented LLM agents on real-world energy analytics tasks, comprising 243 expert-curated problems across market data retrieval, knowledge interpretation, and quantitative modeling.
MLE Agents: Autonomous ML Engineering in Large-scale Recommendation Systems
Meta researchers present MLE Agents, human-on-the-loop LLM agents that autonomously propose, implement, train, and evaluate changes to billion-request-per-day search rankers, running 1,097 experiments with a 29.6% acceptance rate and achieving a 0.23% online search-volume lift, while showing that harness design is as critical as the underlying model.