Feature Generation Using LLMs: An Evolutionary Algorithm Approach

# An Evolutionary Algorithm ApproachThis is the author’s preprint version of a paper later published as: A. Nourbakhsh, B. Alcaraz, and C. Schommer, “Feature Generation Using LLMs: An Evolutionary Algorithm Approach,” in Advances in Explainability, Agents, and Large Language Models, CALM 2024, Communications in Computer and Information Science, vol. 2471, Springer, Cham, 2025. DOI: https://doi.org/10.1007/978-3-031-89103-8_4.
Source: [https://arxiv.org/html/2607.16255](https://arxiv.org/html/2607.16255)
11institutetext:University of Luxembourg, Esch\-sur\-Alzette, Luxembourg
11email:aria\.nourbakhsh@uni\.lu###### Abstract

A crucial step in machine learning pipelines is to present each entity with features or attributes that are representative of the characteristics of the processed entities\. Feature engineering is an important step in finding a relation among attributes that otherwise may not be processed by the ML algorithms\. Meanwhile, Large Language Models have shown promising abilities in coding, mathematical reasoning, and processing world knowledge\. In this work, we utilize an LLM for the problem of feature generation from tabular data based on the previously given features\. We have created a pipeline that takes a set of attributes and a prompt to generate new features\. Then, our selection algorithm selects the best\-performing sets of attributes\. We apply our method to eight datasets from different domains and data types\. Our results show that, in most cases, the language model can produce new features based on mathematical and logical operators that are useful for the given tasks and can improve classification results\.

## 1Introduction

In the field of Machine Learning \(ML\), raw data must be transformed into meaningful features111Or attributes\. Here, we use both terms interchangeably\.to capture relevant patterns effectively\. These attributes describe an entity by numerical, boolean, or string values\. For optimal categorization into different classes or clusters, features must be distinctive and informative, allowing the algorithm to detect hidden patterns across diverse objects\. In other words, ML algorithms induce rules based on how features are represented within the feature space\[[25](https://arxiv.org/html/2607.16255#bib.bib25)\]\. The importance of features cannot be overstated\. These features capture the underlying structure of data\. The most powerful and state\-of\-the\-art algorithms depend on the quality of these features to perform well\.

To extract these features from raw data, one needs to create, select, and modify them based on the information that can be retrieved from the entities of the problem\. This process is time\-consuming and costly as it requires human labor\. An approach to mitigate the issue is to generate attributes based on some of the attributes that have already been extracted for representation\[[23](https://arxiv.org/html/2607.16255#bib.bib23)\]\.

Given a base set of features, another issue is that some of the ML algorithms, such as linear regression, treat each attribute separately\. In this process, a hidden relation among the extracted features can be neglected, and hence, we would miss some crucial information\. Those relations are often limited by logical and arithmetic operations between the result of comparisons between the value of an attribute and a target value \(e\.g\., ‘s​i​z​e=b​i​gsize=big’, or ‘s​i​z​e≥3\.5size\\geq 3\.5’\)\. For instance, if the pattern to identify corresponds to the sum of two numeric attributes being equal to five, it is hard to create a propositional logic formula being true on this condition when the attribute ‘sum’ is not represented in the data, and this is even less scalable \(e\.g\., for two attributesaaandbb, you may create a rule as\(a=0∧b=5\)∨\(a=1∧b=4\)∨…\(a=0\\land b=5\)\\lor\(a=1\\land b=4\)\\lor\.\.\., but this formula, already containing six terms in its disjunctive normal form, would become even longer if the sums were depending on three attributes\)\.

Given the number of features for a dataset, a combination of these features with logical and mathematical operators, such as mean or addition, comprises a vast search space, making it not feasible to look for such relations in a brute\-force manner\. Moreover, the combination of features is unbounded, as one can combine the newly generated features indefinitely\. Some of the approaches to tackle the feature generation problem based on the given features are mentioned in the section[2](https://arxiv.org/html/2607.16255#S2)\.

Meanwhile, we see significant advancements in the field of AI due to the emergence of Large Language Models \(LLMs\)\. Every day, newer capabilities of these models are explored, and they are applied to different problems\. These models are trained on a vast amount of data, and they have the ability to understand, generate, and manipulate human languages with capabilities in solving logical and mathematical problems\[[6](https://arxiv.org/html/2607.16255#bib.bib6)\]\. More importantly, they have shown their capabilities in generating codes in different programming languages\[[8](https://arxiv.org/html/2607.16255#bib.bib8),[21](https://arxiv.org/html/2607.16255#bib.bib21)\]\.

Recently, the scientific community has been exploring the use of LLMs in competitive generation frameworks, where multiple outputs compete to produce superior results\[[24](https://arxiv.org/html/2607.16255#bib.bib24),[39](https://arxiv.org/html/2607.16255#bib.bib39)\]\. This approach, which involves refining outputs iteratively to discover new functions and optimize results, has become central to thefunsearchmethod\[[30](https://arxiv.org/html/2607.16255#bib.bib30)\], pushing the boundaries of search space exploration\. Building upon the funsearch approach, in this paper, we apply a relatively small language model \(LLaMA3\.1 7B\[[11](https://arxiv.org/html/2607.16255#bib.bib11)\]\) to the problem of feature engineering based on the existing features of a dataset\. We experiment with different settings, such as keeping the newly generated features in the prompt, ignoring them, and anonymizing them in a given prompt\. Then, we query the language model with the prompt to create functions to be applied to a given dataset to produce new attributes\. A selection algorithm selects the better\-performing sets of attributes\. We apply this method to eight different datasets with two classification algorithms\. The results show improvement in most of the settings and datasets\.

The advantage of this approach is to a\) bypass the combination of all the relations in the possible search spaces and leverage the production of functions by an LLM to compute them\. b\) Producing transparent features with a function that could be intuitively interpretable\. c\) Diverse sets of features can be produced by this approach that can be scalable to other tasks and datasets\. d\) It can take an arbitrary number of features into account, i\.e\., we do not restrict the feature generation based on a certain number of features\. It helps the ML algorithm to solve problems in datasets such asMonk\-2\(see section[4\.1](https://arxiv.org/html/2607.16255#S4.SS1)\), where the final label is a product of a relation between three attributes\. We show that the LLM can take an arbitrary number of features and, without giving any predefined operator, still produce features that improve the predictability of the model\. In this process, we use a feature selection algorithm inspired by the genetic algorithm to keep and rank the highest and best set of attributes while still exploring less successful sets of features to avoid being stuck in local optima\.

## 2Related Work

In this section, we explore some of the related concepts, such as feature engineering and prompting, and we look at the state\-of\-the\-art approaches to using LLMs in feature engineering for tabular datasets\.

### 2\.1Feature Engineering

All the modern data\-driven approaches of machine learning, such as Decision Trees \(DT\), Neural Networks, and linear classifiers \(e\.g\., Support Vector Machines\), take a given data point as a vector of features\[[4](https://arxiv.org/html/2607.16255#bib.bib4)\]\. These features can consist of various common data types used in computer science to represent entities, such as numerical values, boolean flags, or categorical strings\. Effective feature extraction and selection techniques enhance the representation of the underlying data structure, improving the performance and generalization of downstream tasks or algorithms\[[25](https://arxiv.org/html/2607.16255#bib.bib25),[9](https://arxiv.org/html/2607.16255#bib.bib9),[33](https://arxiv.org/html/2607.16255#bib.bib33)\]\. Feature generation and engineering depend on factors like the availability of extractable information and domain knowledge, which can be resource\-intensive\. However, new features can often be derived from existing ones\[[7](https://arxiv.org/html/2607.16255#bib.bib7),[13](https://arxiv.org/html/2607.16255#bib.bib13)\]\. For instance, Body Mass Index \(BMI\) is calculated as the ratio of weight to height\[[13](https://arxiv.org/html/2607.16255#bib.bib13)\], or Total Sales Revenue is the product ofu​n​i​t​\_​p​r​i​c​e×q​u​a​n​t​i​t​y​\_​s​o​l​dunit\\\_price\\times quantity\\\_sold\.

This type of feature construction is important for better classification performance\[[10](https://arxiv.org/html/2607.16255#bib.bib10)\]\. Many approaches to this type of feature construction have been developed through the years, such as greedy strategy\[[31](https://arxiv.org/html/2607.16255#bib.bib31)\], applying polynomial functions\[[32](https://arxiv.org/html/2607.16255#bib.bib32)\], and using arithmetic and logical operators to construct new feature\[[23](https://arxiv.org/html/2607.16255#bib.bib23)\]\. For example, in\[[10](https://arxiv.org/html/2607.16255#bib.bib10)\], they apply common mathematical functions and arithmetic operators by randomly choosing an operator and applying them to pairs of features\. Then, an algorithm keeps the best\-performing sets of features over many iterations\.

Alternative approaches, such as Reinforcement Learning, have been employed to optimize feature spaces\[[17](https://arxiv.org/html/2607.16255#bib.bib17)\], and tree traversal techniques have been utilized for feature generation and selection\[[18](https://arxiv.org/html/2607.16255#bib.bib18)\]\. It is also possible to combine a pair or more number of features to create a new one\[[16](https://arxiv.org/html/2607.16255#bib.bib16)\]\.

### 2\.2Prompting, Feature Selection, and Generation

Prompting using LLMs is a new trend that exploits knowledge by predicting the probability of word sequences\. These language models show incredible performance in reasoning\[[37](https://arxiv.org/html/2607.16255#bib.bib37),[2](https://arxiv.org/html/2607.16255#bib.bib2)\]and domain\-specific applications such as medicine\[[15](https://arxiv.org/html/2607.16255#bib.bib15)\]and finance\[[20](https://arxiv.org/html/2607.16255#bib.bib20)\]\. However, these models have some shortcomings\. Since they are trained on data from the internet and rely on language modeling to predict the most probable tokens, they are prone to hallucinations and generating biased outputs\[[27](https://arxiv.org/html/2607.16255#bib.bib27)\]\.

We identified four relevant studies that utilize LLMs for feature selection and feature engineering\.\[[14](https://arxiv.org/html/2607.16255#bib.bib14)\]applies LLMs to feature selection, using various prompts and strategies for selecting, ranking, and incrementally choosing features\. Similarly,\[[19](https://arxiv.org/html/2607.16255#bib.bib19)\]employs prompt\-based techniques for feature selection and shows that the feature importance identified by LLMs correlates strongly with traditional feature importance estimation methods\. Their approach incorporates feature values alongside target labels\. In another setting, they provide the data and feature description to the LLM for feature selection\.

For feature engineering\[[12](https://arxiv.org/html/2607.16255#bib.bib12)\]applies a prompt to generate features by a limited set of mathematical operators\. They show promising results by providing the task description, features, and examples of training data\. They explicitly ask the LLM about analyzing the relevance of the features and the task at hand\. Further, they prompt by asking to create ten binary features by the use of ‘is in’, ‘≥\\geq’, and ‘≤\\leq’ keywords\. Also, they explicitly ask about the possible range of values from the LLM\. Notably, they use a limited set of operators and more complex, longer prompts\.

The work in ‘Dynamic and Adaptive Feature Generation with LLM’\[[40](https://arxiv.org/html/2607.16255#bib.bib40)\]is the most closely related approach to our work\. In their method, a feature set and a predefined set of mathematical operators are provided to a pipeline by prompting an LLM\. Then, in prompting, they combine original and newly generated features\. This is followed by an iterative feature selection process, which evaluates the generated features on downstream tasks and retains the optimal dataset\. The process employs a set of predefined unary and binary mathematical operators\.

Our approach differs from the work above in several ways: a\) Unlike\[[40](https://arxiv.org/html/2607.16255#bib.bib40)\], we do not restrict the LLM to a predefined set of operators or binary feature combinations\. Instead, our method allows the creation of new features using any number of related features, with outputs that can be boolean, string, or real numbers\. This contrasts with\[[12](https://arxiv.org/html/2607.16255#bib.bib12)\], where output functions are limited to binary values\. b\) We demonstrate that the LLM can generate new features with a relatively short prompt\. c\) We show that even with minimal information about the dataset, the LLM can produce new features that enhance classification performance\. d\) Our results indicate that competitive performance can be achieved using a local LLM that runs on a personal computer, with only a short prompt describing the intended output and minimal information about the data\. e\) To mitigate the influence of biased knowledge within LLMs, we propose anonymizing the input and relying solely on the LLM’s ability to generate well\-formed, viable functions without any domain\-specific knowledge\.

## 3Methodology

This section describes the pipeline and the prompting command we devised for our approach\.

### 3\.1Pipeline

Our pipeline222Code is available on the GitHub page: https://github\.com/zaap38/funsearch\-attis composed of three modules: 1\) A feature generator module powered by an LLM, 2\) An evaluator that applies a machine learning algorithm to learn from and assess the generated features, and 3\) A selection algorithm that identifies the top\-performing feature sets to refine and further apply the generated features\. The pseudocode for this pipeline is outlined in Algorithm[1](https://arxiv.org/html/2607.16255#algorithm1)\.

The pipeline gets the raw dataset, The number of iterationsnn, and a maximum sample countMM\. Each sample is a set of attributes\. The output of the pipeline is a set of refined dataset samples along with their F1\-measures\. First, the algorithm calculates the F1 of the initial datasetDD, and then a set of samples is initialized, containing the original datasetDDand its associated F1\-measure\. The algorithm then enters a loop that will run fornniterations\. It first sums the F1s of all current samples\. It then selects one sample randomly, with a higher probability of selection for samples with higher F1\. This is done by generating a random number between 0 and the sum of all the F1s and progressively subtracting each sample’s F1 from this random number until it becomes less than or equal to zero, indicating the selected sample\.

Once a sample is selected, the algorithm generates a mutation prompt by callinggetPrompton the selected dataset\. This prompt is then passed to an LLM via askLLM method to generate a new attribute\. The new attribute is added to the selected sample, and the F1 of this mutated dataset is recalculated\. The mutated dataset and its new F1 are added to the list of samples\. The LLM generates both an attribute name and a Python expression, formatted in a way that can be directly applied to a Pandas DataFrame structure \(see section[3\.2](https://arxiv.org/html/2607.16255#S3.SS2)\) We iteratively prompt the LLM to create new features using the above mentioned evolutionary process\.

After mutating and adding the new sample, the algorithm cleans up the sample set\. It creates a new set of samples by randomly selecting up toMMsamples from the current set, again using the probability\-based selection method \(where F1\-measure samples have a higher chance of being selected\)\. After selectingMMsamples, the old sample set is replaced with the new, cleaned\-up set\. The loop repeats this process fornniterations, refining the dataset samples and their associated accuracies\. After the loop finishes, the algorithm outputs the final set of samples, each with its associated F1, and we select the highest\-scoring sample as the selected dataset\.

Input:Dataset

DD, Number of iterations

nn, Maximum sample count

MM
Output:A set of refined dataset samples with their F1s

F​1←getF1​\(D\)F1\\leftarrow\\texttt\{getF1\}\(D\)

s​a​m​p​l​e​s←\{\(D,F​1\)\}samples\\leftarrow\\\{\(D,F1\)\\\}
for*i←1i\\leftarrow 1tonn*do

s​u​m←0sum\\leftarrow 0
foreach*s​a​m​p​l​e∈s​a​m​p​l​e​ssample\\in samples*do

s​u​m←s​u​m\+s​a​m​p​l​e​\[1\]sum\\leftarrow sum\+sample\[1\]
end foreach

r​a​n​d←random​\(0,s​u​m\)rand\\leftarrow\\texttt\{random\}\(0,sum\)
foreach*s​a​m​p​l​e∈s​a​m​p​l​e​ssample\\in samples*do

r​a​n​d←r​a​n​d−s​a​m​p​l​e​\[1\]rand\\leftarrow rand\-sample\[1\]
if*r​a​n​d≤0rand\\leq 0*then

d←s​a​m​p​l​e​\[0\]d\\leftarrow sample\[0\]
break

end if

end foreach

m​s​g←getPrompt​\(d\)msg\\leftarrow\\texttt\{getPrompt\}\(d\)
a​t​t​r​i​b​u​t​e←askLLM​\(m​s​g\)attribute\\leftarrow\\texttt\{askLLM\}\(msg\)
d←addAttribute​\(d,a​t​t​r​i​b​u​t​e\)d\\leftarrow\\texttt\{addAttribute\}\(d,attribute\)
F​1←getF1​\(d\)F1\\leftarrow\\texttt\{getF1\}\(d\)
s​a​m​p​l​e​s\.add​\(\(d,F​1\)\)samples\.\\texttt\{add\}\(\(d,F1\)\)
n​e​w​\_​s​a​m​p​l​e​s←∅new\\\_samples\\leftarrow\\emptyset
for*j←1j\\leftarrow 1toMM*do

s​u​m←0sum\\leftarrow 0
foreach*s​a​m​p​l​e∈s​a​m​p​l​e​ssample\\in samples*do

s​u​m←s​u​m\+s​a​m​p​l​e​\[1\]sum\\leftarrow sum\+sample\[1\]
end foreach

r​a​n​d←random​\(0,s​u​m\)rand\\leftarrow\\texttt\{random\}\(0,sum\)
foreach*s​a​m​p​l​e∈s​a​m​p​l​e​ssample\\in samples*do

r​a​n​d←r​a​n​d−s​a​m​p​l​e​\[1\]rand\\leftarrow rand\-sample\[1\]
if*r​a​n​d≤0rand\\leq 0*then

n​e​w​\_​s​a​m​p​l​e​s\.add​\(s​a​m​p​l​e\)new\\\_samples\.\\texttt\{add\}\(sample\)
break

end if

end foreach

end for

s​a​m​p​l​e​s←n​e​w​\_​s​a​m​p​l​e​ssamples\\leftarrow new\\\_samples
end for

Algorithm 1Pseudo\-code for dataset mutation and selection
### 3\.2Prompting

At the heart of our pipeline, there is the LLM that gets a prompt and produces a Pythonic expression applicable to a Pandas Dataframe\[[34](https://arxiv.org/html/2607.16255#bib.bib34)\]\. We use the following prompt shown in Figure[1](https://arxiv.org/html/2607.16255#S3.F1)to instruct the LLM on the task it needs to perform\. Note that the LLM is not aware of the previous prompts \(i\.e\., we don’t have a memory of the previous prompts for the LLM\)\.

Take the following dataframe attributes, their dtype, and possible values\. Create exactly one new attribute based on them using mathematical and logical operators\. You can use numpy and math libraries if needed\. return the result without code format and without extra text as below:n: new\_attribute\_name a short expression that can be applied to a dataframe

df\[feature\_name\_1\] : dtype\-possible values \[\.\.\.\] \. \. \. df\[feature\_name\_n\] : dtype\-possible values \[\.\.\.\]

Figure 1:The prompt used to generate a Python code to create a function applicable to a Pandas dataframe\.In the prompt, we provide the original feature names and their data type\. For categorical and boolean values, as long as they are fewer than 20 values, we enumerate their possible values\. For the attributes that contain integers or floating point values, we give the minimum and maximum range of that column to the prompt\.

In our experiments, we apply different settings: a\) We retain the newly generated feature sets and provide the LLM with both the original and newly generated features\. The advantage of this approach is the potential to discover valuable features derived from the newly created ones\. However, for large datasets, this can increase the model’s computational complexity and running time\. Also, having more features can lead to the ‘curse of dimensionality’\[[3](https://arxiv.org/html/2607.16255#bib.bib3)\]\.

b\) The model ignores the newly generated features, creating only one new attribute based on the original dataset\. This approach reduces evaluation time and minimizes the risk of introducing noise into the feature sets\. It also reduces the chance of the LLM creating malformed functions\.

c\) Is the same as \(b\); however, we anonymize the feature names and values so the LLM cannot use any learned knowledge or correlations to build new features\.

More formally, our approach can be expressed as follows: LetAAbe all the possible attributes\. We define the functionP:2A→ΣℕP:2^\{A\}\\rightarrow\\Sigma^\{\\mathbb\{N\}\}\(whereΣ\\Sigmais the alphabet containing all the ASCII characters, and soΣℕ\\Sigma^\{\\mathbb\{N\}\}is the formal definition of a string\), which from the list of the attributes of a dataset, returns a string which we will use as our prompt for the LLM\. Consequently, letVVbe all the dataset rows andViV\_\{i\}be one row\. We define the LLM as a functionL:Σℕ→ℳL:\\Sigma^\{\\mathbb\{N\}\}\\rightarrow\\mathcal\{M\}whereℳ:Vi→Vi∪Y\\mathcal\{M\}:V\_\{i\}\\rightarrow V\_\{i\}\\cup YwithY∈\{ℝ,Σℕ\}Y\\in\\\{\\mathbb\{R\},\\Sigma^\{\\mathbb\{N\}\}\\\}\(meaning it can be a boolean, a real, or a string\) andVi∈VV\_\{i\}\\in V\.

In scenario \(a\),AAis defined as the set of both the original attributes in the dataset and the newly generated ones\. In contrast, in scenario \(b\),AArefers solely to the base set of original attributes\.

## 4Evaluation

This section presents an overview of the datasets and experimental setup, followed by an analysis of the results\.

### 4\.1Data

As mentioned in the introduction, the search space for new attributes can be unbounded\. While this may initially seem arbitrary, we envision that an LLM could help narrow the search space by being provided with minimal information about a dataset to produce a mathematical or logical function\. For this reason, we chose two types of datasets from previous work and publicly available datasets\. The first group of these datasets has rich feature names, so the LLM has access to the context and semantics of the datasets\. The second group has three datasets whose feature names do not provide any information about the task\.

In a separate experimental setup, for comparison, we also anonymize the first group of the datasets, where each attribute is transformed into ‘attribute\_n’, with ‘n’ denoting the attribute’s position in the dataset\. Additionally, categorical values are converted into numerical representations\. We expect that when the LLM is provided with semantic context of the features, it may be able to generate more insightful new attributes, albeit with potential biases\.

We take three out of four datasets used by\[[40](https://arxiv.org/html/2607.16255#bib.bib40)\]to compare our approaches\. One of their datasets is the Amazon Commerce Review\[[22](https://arxiv.org/html/2607.16255#bib.bib22)\], which we think is inappropriate for this task as it fits other Natural Language Processing techniques better because each column represents a letter with more than 10000 features\. One of the other three datasets is Ionosphere \(Ion\)\[[36](https://arxiv.org/html/2607.16255#bib.bib36)\]\. It has 34 features that represent a phased array of 16 high\-frequency antennas\. As a result, all the features have numerical values, and the labels are binary forgoodandbadsignals\.

Another dataset is the Diabetes Health Indicators Dataset \(Dia\)\[[35](https://arxiv.org/html/2607.16255#bib.bib35)\]\. We take a subset of this dataset\. The dataset has 22 health\-related features with two labels indicating whether or not a person has diabetes\. The dataset has a mix of boolean and real numbers for feature values\.

Abalone is another dataset from the previous work that we used for the experiments\[[26](https://arxiv.org/html/2607.16255#bib.bib26)\]\. Abalone \(Aba\) has eight features for predicting the age of abalones\. These features consist of length, diameter, and other indicators of abalones’ characteristics333This dataset is made for predicting the age of abalones, which has 30 different values\. However, in\[[40](https://arxiv.org/html/2607.16255#bib.bib40)\], they indicate that it is a binary prediction, which is different from the original description of data\.\.

In addition to the datasets mentioned above, we selected additional datasets spanning various tasks and domains to ensure a broader and more diverse evaluation\. The first one, Monk\-2\[[38](https://arxiv.org/html/2607.16255#bib.bib38)\], is a synthetic dataset consisting of six numerical attributes with integer values ranging from11to44, and two classes \(0and11\)\. In the specific case of Monk\-2, the class has the value11if and only if exactly two of the six attributes have the value11\. More formally:

class=\{1if​\|\{a∈Attributes∣avalue=1\}\|=20otherwise\\text\{class\}=\\begin\{cases\}1&\\text\{if \}\|\\\{a\\in\\text\{Attributes\}\\mid a\_\{\\text\{value\}\}=1\\\}\|=2\\\\ 0&\\text\{otherwise\}\\end\{cases\}
This dataset is interesting as the class is not bounded to a simple equality/inequality between some attributes and a constant\. There exists a trivial propositional formula representing this condition, but it is large enough so it cannot be easily represented with a predefined list of operators containing only the boolean and arithmetic operators or basic functions such asx2x^\{2\},l​o​g2​\(x\)log\_\{2\}\(x\), orx\\sqrt\{x\}\.

Monk\-3\[[38](https://arxiv.org/html/2607.16255#bib.bib38)\]is similar to Monk\-2, but it follows a more complex relationship, and the labels are determined by the following expression:\(a​5=3∧a​4=1\)∨\(a​5≠4∧a​2≠3\)\(a5=3\\land a4=1\)\\lor\(a5\\neq 4\\land a2\\neq 3\)wherea​nanis a column in the dataset\.

Another dataset is named Car dataset\[[5](https://arxiv.org/html/2607.16255#bib.bib5)\]\. It classifies a car in terms of buying acceptability \(Unacceptable, Acceptable, Good, or Very good\) based on several attributes, such as the number of seats or the size of the luggage boot\. This dataset is interesting because it is simple to run a qualitative evaluation of the generated attributes \(As it is easy to understand why a low buying price or a large number of seats pushes the acceptability level toward Very good\)\.

Finally, we use Wine\[[1](https://arxiv.org/html/2607.16255#bib.bib1)\]and Predict Students’ Dropout and Academic Success \(Stu\) datasets\[[29](https://arxiv.org/html/2607.16255#bib.bib29)\]with 13 and 36 features respectively\. Stu dataset contains a diverse set of personal, educational, and demographic attributes\. The Wine dataset consists of attributes that describe a wine characteristic\. These datasets represent distinct domains and feature a variety of attributes and data types, making them well\-suited for evaluating our pipeline and the capabilities of the LLM\.

Table 1:Datasets size and number of features\.Table[1](https://arxiv.org/html/2607.16255#S4.T1)summarizes the number of samples and features in each dataset\. The datasets on the right side of the bold line are without meaningful feature names\.

### 4\.2Results

This section presents the results of our classification pipeline using the Multi\-Layer Perceptron \(MLP\) and Decision Tree \(DT\) algorithms\. We report four outcomes for each algorithm based on the methodology described in section[3](https://arxiv.org/html/2607.16255#S3)\. First, as a baseline, we provide the results of running the algorithms on the raw, original attributes, referred to asRawmethod\. Next, the results for aggregating both base and generated attributes for prompting are reported under the nameAgg\. In the third scenario, we use only the base attributes for feature generation, also named theSkipmethod\. Lastly, the anonymization of attributes and their values is presented under the labelAnon\.

In Table[2](https://arxiv.org/html/2607.16255#S4.T2), we separate the three datasets at the bottom of the table that, by design, are anonymous, with feature names that do not carry any semantic meaning related to the task\. For our experiments, we tested various local open\-source language models and selected LLaMA3\.1 7B\[[11](https://arxiv.org/html/2607.16255#bib.bib11)\]due to its ability to consistently provide better, more diverse, and more parsable functions\. All experiments were conducted on a single NVIDIA GeForce 4090 using the scikit\-learn\[[28](https://arxiv.org/html/2607.16255#bib.bib28)\]implementation of the aforementioned algorithms\. Table[2](https://arxiv.org/html/2607.16255#S4.T2)presents the outcomes ofn=400n=400iterations of the pipeline, with an early stopping set to the patience of 20 queries \(i\.e\., if 20 consecutive queries to the LLM do not improve the result on the validation set, the process stops, and the best\-performing sample is returned\)\. We split the data into 70% for training, with the remaining 30% equally divided between development and testing sets\. Following standard supervised learning practices, we optimize the pipeline on the development set and report the results on the test set\. Due to the stochastic nature of the LLMs and our pipeline, we report the average result on the test sets and the final best\-performing attributes over three runs\.

Table 2:The F1 result of our evolutionary pipeline using LLM feature generation and selection algorithm\.Rawis the baseline without generated features\. InAgg, we use generated features in the prompt, while in theSkip, only the base features are used\.Anonis the same asSkip, but the features are anonymized\. The three bottom datasets are anonymous by design\. We report the change in results with the best\-performing setting compared to the baseline\.For the DT algorithm, the Aba dataset gets the highest change for the datasets at the top of the table\. This is achieved inAnonymsetting where the description of the dataset is anonymized\. Moreover, we see a great improvement for the anonymous datasets at the bottom of the table, suggesting a great potential for LLMs to produce functions without access to the semantics of the dataset\. In particular, for Monk datasets, we see a big jump from the baseline\. This shows that the produced attributes could take a ternary relation into account\. In other cases, where the model had access to the feature names, keeping the generated features in the feature generation queries has resulted in better scores\.

For the MLP algorithm, feature aggregation \(Agg\) and feature skipping \(Skip\), in most cases, improve the performance\. This is especially apparent in datasets like Car, Wine and Monk\-2, where theAggandSkipconfigurations yield significant performance boosts\. Notably, the Stu and Monk\-2 datasets still exhibit strong performance even in the anonymized setting, indicating that the algorithms can maintain robustness despite removing semantic information from the feature names\.

We were expecting that keeping the generated features to produce new samples \(i\.e\., prompting with the aggregated features\) would degrade the quality of the intake and generation of the new features\. As we can see for MLP, this behavior is random and based on the produced features; in some cases, theSkipmethod scores better thanAggand vice versa\. Overall, in 13 out of 16 experiments with DT and MLP, our evolutionary feature generation increases the performances of the results\.

Unfortunately, we could not reproduce the results of\[[40](https://arxiv.org/html/2607.16255#bib.bib40)\], even for the base cases without additional features\. This discrepancy may stem from differences in random sampling or the splitting of data into training, development, and test sets\. Consequently, a direct comparison between our results and theirs is not feasible\. Nonetheless, for the Dia dataset, our approach demonstrates a more significant improvement between the raw data and our best\-performing model\. Specifically, using the DT algorithm, we achieved a 1\.93% increase in F1 measure \(from 64\.38% to 66\.31%\), in contrast to the marginal decline reported by Zhang et al\. \(from 59\.7% to 60%\)\.

## 5Discussion

In this section, we look at some insights and behavior of the LLM and the implemented algorithm for optimization\.

### 5\.1Operators

The analysis of the number of operators produced in all runs and datasets, as shown in Figure[2](https://arxiv.org/html/2607.16255#S5.F2), reveals that the language model tends to produce arithmetic operators more frequently\. Addition \(\+\+\), division \(//\), and multiplication \(\*\) are the most commonly generated operators for the functions\. Surprisingly, the model has generated more ‘&\\&’ symbols than the keyword ‘a​n​dand’\. Additionally, there is a preference for using the numpy package over Python’s math library\.

Logical and comparison operators, such as equality \(==\), greater than \(\>\), and logical conjunctions \(and, or\), exhibit significant frequencies, emphasizing their importance in conditional statements and control flows within the generated code\. These findings underscore the model’s capability to generate code that is not only mathematically robust but also aligned with common programming practices in data science and numerical analysis\. However, to force the model to produce arithmetic and mathematical operators, in the prompt \(Figure[1](https://arxiv.org/html/2607.16255#S3.F1)\), we explicitly specified, ‘You can use numpy and math library if needed\.’ Otherwise, the model tends to use operators with boolean outcomes\.

![Refer to caption](https://arxiv.org/html/2607.16255v1/x1.png)Figure 2:Figure shows the total number of generated operators in all the final created sets of attributes for all the datasets and runs\.
### 5\.2Number of operators per function

Another interesting measure would be to see how many operators the LLM has generated to generate a new feature\. We take theAggandSkipmethods and visualize the distribution in Figure[3](https://arxiv.org/html/2607.16255#S5.F3)\. The LLM utilizes a broad range of operators, varying from 2 to 50 per function\. However, the average number of operators typically falls within the range of 4 to 7\. Moreover, we see more and broader range of operators were generated for the Car and Dia datasets\. It is an unexpected behavior as the number of base features of these two datasets is noticeably different\.

We also conducted Mann\-Whitney U tests to compare the number of operators per generated function between theAggandSkipmethods\. In the unadjusted analyses, Dia and Monk\-2 showed nominally significant differences \(p=0\.0071p=0\.0071andp=0\.0021p=0\.0021, respectively\)\. The early\-stopping mechanism in our pipeline limited the accumulation of generated attributes\. In experiments without this constraint, as the number of available attributes increased, the LLM tended to generate longer and more complex expressions\. These expressions were more likely to fail during Python parsing\.

![Refer to caption](https://arxiv.org/html/2607.16255v1/x2.png)Figure 3:The boxplot of the distribution of operators per generated functions\. In some cases, we see functions with more than 20 operators\.
### 5\.3Running time

Figure[4](https://arxiv.org/html/2607.16255#S5.F4)shows the average running time of each method on the datasets\. Running time is subjective and depends on hardware, operating system, and other uncontrollable factors\. However, on a computer with AMD Ryzen 7900X and Nvidia RTX4090 GPU, we report the average running time of the pipeline over three runs\. Also, a model may have converged earlier than the others\. According to Figure[4](https://arxiv.org/html/2607.16255#S5.F4), the trend shows that, as expected, the larger dataset \(Dia\) took more time to be processed with our pipeline, while smaller datasets like Monk\-2 and Monk\-3 were processed more quickly\. Also, generally speaking, running the pipeline with the MLP algorithm takes more time than the DT, adding a computational bottleneck to evaluate the features on the downstream tasks\. On three datasets, namely Aba and Dia, keeping the generated features\(Aggmodels, green bars\) has led to a faster convergence444in our experiments, we tried running the pipeline without the early stopping\. In the case ofAgg, feeding the LLM with newly generated features leads to more error\-prone generation of functions\.\.

![Refer to caption](https://arxiv.org/html/2607.16255v1/x3.png)Figure 4:Figure shows the average and standard deviation of the running time of each approach on each dataset\. Please pay attention that the last three datasets are anonymous datasets by design\.

## 6Conclusion

In this paper, we took the concept of funsearch\[[30](https://arxiv.org/html/2607.16255#bib.bib30)\]and applied it to the feature generation problem\. Namely, we took eight datasets and asked an LLM to generate novel features based on the given attributes\. We created three settings: a\) We kept the base attributes and added the newly generated ones to the prompt\. b\) We skipped the newly generated attributes in prompting the LLM, and c\) We did the same as \(b\) with anonymizing the feature names and possible value ranges\. On top of that, We devised an evolutionary algorithm inspired by genetic algorithm, which selects the best\-generated sets of attributes\.

In this process, we showed that we could reach a fast convergence by utilizing a relatively small local language model that, in most cases, improves the baseline\. More importantly, we showed that in the cases where the LLM has no access to contextual knowledge by anonymizing the description of the data, it is still capable of producing functions that may improve the result\. This is important because we do not rely on biased knowledge of LLM, and the model only searches for the possible space of the functions to produce new attributes\. In other words, our pipeline relied only on the code\-generation ability of the LLM\.

## 7Limitations and future work

Our approach has some limitations worth noting\. First, we use a small LM\. Better results and more parsable functions are expected to be obtained from using a larger LM or a commercial one\.

The whole pipeline and the stochastic nature of the LLMs, make it hard to reproduce the result of each run of the feature generation\. Moreover, LLMs are trained on human data, and they are biased toward the training data artifacts\. Our suggestion to bypass this problem was to anonymize the description of data in the given prompt\. However, it is worth noting that the code\-generation ability of the LLM may also be biased, and the LLM tends to generate one operator more commonly than the others\. One potential solution to this problem is to explicitly ask the LLM which operators it can use or to specify the returned data types of the function\.

\{credits\}

#### 7\.0\.1Acknowledgements

We thank the Luxembourg National Research Fund \(FNR\) for the funding of this research as part of the project C21 \- Collaboration 21: IPBG2020/IS/14839977/C21\.

#### 7\.0\.2\\discintname

None of the author of the paper is having a conflict of interest\.

## References

- \[1\]Aeberhard, S\., Forina, M\.: Wine\. UCI Machine Learning Repository \(1992\), DOI: https://doi\.org/10\.24432/C5PC7J
- \[2\]Ahn, J\., Verma, R\., Lou, R\., Liu, D\., Zhang, R\., Yin, W\.: Large language models for mathematical reasoning: Progresses and challenges\. arXiv preprint arXiv:2402\.00157 \(2024\)
- \[3\]Bellman, R\.: Dynamic programming\. Princeton University Press, Princeton \(1957\)
- \[4\]Bengio, Y\., Courville, A\., Vincent, P\.: Representation learning: A review and new perspectives\. IEEE Transactions on Pattern Analysis and Machine Intelligence35\(8\), 1798–1828 \(2013\)\. https://doi\.org/10\.1109/TPAMI\.2013\.50
- \[5\]Bohanec, M\.: Car Evaluation\. UCI Machine Learning Repository \(1997\), DOI: https://doi\.org/10\.24432/C5JP48
- \[6\]Chang, Y\., Wang, X\., Wang, J\., Wu, Y\., Yang, L\., Zhu, K\., Chen, H\., Yi, X\., Wang, C\., Wang, Y\., et al\.: A survey on evaluation of large language models\. ACM Transactions on Intelligent Systems and Technology15\(3\), 1–45 \(2024\)
- \[7\]Coates, A\., Ng, A\., Lee, H\.: An analysis of single\-layer networks in unsupervised feature learning\. In: Proceedings of the fourteenth international conference on artificial intelligence and statistics\. pp\. 215–223\. JMLR Workshop and Conference Proceedings \(2011\)
- \[8\]Coello, C\.E\.A\., Alimam, M\.N\., Kouatly, R\.: Effectiveness of chatgpt in coding: a comparative analysis of popular large language models\. Digital4\(1\), 114–125 \(2024\)
- \[9\]Dash, M\., Liu, H\.: Feature selection for classification\. Intelligent data analysis1\(1\-4\), 131–156 \(1997\)
- \[10\]Dor, O\., Reich, Y\.: Strengthening learning algorithms by feature discovery\. Information Sciences189, 176–190 \(2012\)
- \[11\]Dubey, A\., et al\.: The llama 3 herd of models \(2024\),[https://arxiv\.org/abs/2407\.21783](https://arxiv.org/abs/2407.21783)
- \[12\]Han, S\., Yoon, J\., Arik, S\.O\., Pfister, T\.: Large language models can automatically engineer features for few\-shot tabular learning\. In: Salakhutdinov, R\., Kolter, Z\., Heller, K\., Weller, A\., Oliver, N\., Scarlett, J\., Berkenkamp, F\. \(eds\.\) Proceedings of the 41st International Conference on Machine Learning\. Proceedings of Machine Learning Research, vol\. 235, pp\. 17454–17479\. PMLR \(21–27 Jul 2024\),[https://proceedings\.mlr\.press/v235/han24f\.html](https://proceedings.mlr.press/v235/han24f.html)
- \[13\]Heaton, J\.: An empirical analysis of feature engineering for predictive modeling\. In: SoutheastCon 2016\. pp\. 1–6\. IEEE \(2016\)
- \[14\]Jeong, D\.P\., Lipton, Z\.C\., Ravikumar, P\.: Llm\-select: Feature selection with large language models\. arXiv preprint arXiv:2407\.02694 \(2024\)
- \[15\]Karabacak, M\., Margetis, K\.: Embracing large language models for medical applications: opportunities and challenges\. Cureus15\(5\) \(2023\)
- \[16\]Katz, G\., Shin, E\.C\.R\., Song, D\.: Explorekit: Automatic feature generation and selection\. In: 2016 IEEE 16th international conference on data mining \(ICDM\)\. pp\. 979–984\. IEEE \(2016\)
- \[17\]Khurana, U\., Samulowitz, H\., Turaga, D\.: Feature engineering for predictive modeling using reinforcement learning\. In: Proceedings of the AAAI Conference on Artificial Intelligence\. vol\. 32 \(2018\)
- \[18\]Khurana, U\., Turaga, D\., Samulowitz, H\., Parthasrathy, S\.: Cognito: Automated feature engineering for supervised learning\. In: 2016 IEEE 16th international conference on data mining workshops \(ICDMW\)\. pp\. 1304–1307\. IEEE \(2016\)
- \[19\]Li, D\., Tan, Z\., Liu, H\.: Exploring large language models for feature selection: A data\-centric perspective\. arXiv preprint arXiv:2408\.12025 \(2024\)
- \[20\]Li, Y\., Wang, S\., Ding, H\., Chen, H\.: Large language models in finance: A survey\. In: Proceedings of the fourth ACM international conference on AI in finance\. pp\. 374–382 \(2023\)
- \[21\]Liu, J\., Xia, C\.S\., Wang, Y\., Zhang, L\.: Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation\. Advances in Neural Information Processing Systems36\(2024\)
- \[22\]Liu, Z\.: Amazon Commerce Reviews\. UCI Machine Learning Repository \(2011\), DOI: https://doi\.org/10\.24432/C55C88
- \[23\]Markovitch, S\., Rosenstein, D\.: Feature generation using general constructor functions\. Machine Learning49, 59–98 \(2002\)
- \[24\]Meyerson, E\., Nelson, M\.J\., Bradley, H\., Gaier, A\., Moradi, A\., Hoover, A\.K\., Lehman, J\.: Language model crossover: Variation through few\-shot prompting\. arXiv preprint arXiv:2302\.12170 \(2023\)
- \[25\]Motoda, H\., Liu, H\.: Feature selection, extraction and construction\. Communication of IICM \(Institute of Information and Computing Machinery, Taiwan\)5\(67\-72\), 2 \(2002\)
- \[26\]Nash, Warwick, Sellers, Tracy, Talbot, Simon, Cawthorn, Andrew, Ford, Wes: Abalone\. UCI Machine Learning Repository \(1995\), DOI: https://doi\.org/10\.24432/C55C7W
- \[27\]Navigli, R\., Conia, S\., Ross, B\.: Biases in large language models: origins, inventory, and discussion\. ACM Journal of Data and Information Quality15\(2\), 1–21 \(2023\)
- \[28\]Pedregosa, F\., Varoquaux, G\., Gramfort, A\., Michel, V\., Thirion, B\., Grisel, O\., Blondel, M\., Prettenhofer, P\., Weiss, R\., Dubourg, V\., Vanderplas, J\., Passos, A\., Cournapeau, D\., Brucher, M\., Perrot, M\., Duchesnay, E\.: Scikit\-learn: Machine learning in Python\. Journal of Machine Learning Research12, 2825–2830 \(2011\)
- \[29\]Realinho, Valentim, Martins, V\., Mónica, Machado, Jorge, Baptista, Luís: Predict Students’ Dropout and Academic Success\. UCI Machine Learning Repository \(2021\), DOI: https://doi\.org/10\.24432/C5MC89
- \[30\]Romera\-Paredes, B\., Barekatain, M\., Novikov, A\., Balog, M\., Kumar, M\.P\., Dupont, E\., Ruiz, F\.J\., Ellenberg, J\.S\., Wang, P\., Fawzi, O\., et al\.: Mathematical discoveries from program search with large language models\. Nature625\(7995\), 468–475 \(2024\)
- \[31\]Shafti, L\.S\., Pérez, E\.P\.: Constructive induction and genetic algorithms for learning concepts with complex interaction\. In: Proceedings of the 7th Annual Conference on Genetic and Evolutionary Computation\. p\. 1811–1818\. GECCO ’05, Association for Computing Machinery, New York, NY, USA \(2005\)\. https://doi\.org/10\.1145/1068009\.1068317,[https://doi\-org\.proxy\.bnl\.lu/10\.1145/1068009\.1068317](https://doi-org.proxy.bnl.lu/10.1145/1068009.1068317)
- \[32\]Sutton, R\.S\., Matheus, C\.J\.: Learning polynomial functions by feature construction\. In: Machine Learning Proceedings 1991, pp\. 208–212\. Elsevier \(1991\)
- \[33\]Tang, J\., Alelyani, S\., Liu, H\.: Feature selection for classification: A review\. Data classification: Algorithms and applications p\. 37 \(2014\)
- \[34\]pandas development team, T\.: pandas\-dev/pandas: Pandas \(feb 2020\)\. https://doi\.org/10\.5281/zenodo\.3509134,[https://doi\.org/10\.5281/zenodo\.3509134](https://doi.org/10.5281/zenodo.3509134)
- \[35\]Teboul, A\.: Diabetes health indicators dataset \(2022\),[https://www\.kaggle\.com/datasets/alexteboul/diabetes\-health\-indicators\-dataset](https://www.kaggle.com/datasets/alexteboul/diabetes-health-indicators-dataset)
- \[36\]V, S\., S\., W\., L\., H\., K\., B\.: Ionosphere\. UCI Machine Learning Repository \(1989\), DOI: https://doi\.org/10\.24432/C5W01B
- \[37\]Wang, Z\.: Causalbench: A comprehensive benchmark for evaluating causal reasoning capabilities of large language models\. In: Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing \(SIGHAN\-10\)\. pp\. 143–151 \(2024\)
- \[38\]Wnek, J\.: MONK’s Problems\. UCI Machine Learning Repository \(1992\), DOI: https://doi\.org/10\.24432/C5R30R
- \[39\]Wu, X\., Wu, S\.h\., Wu, J\., Feng, L\., Tan, K\.C\.: Evolutionary computation in the era of large language model: Survey and roadmap\. arXiv preprint arXiv:2401\.10034 \(2024\)
- \[40\]Zhang, X\., Zhang, J\., Rekabdar, B\., Zhou, Y\., Wang, P\., Liu, K\.: Dynamic and adaptive feature generation with llm\. arXiv preprint arXiv:2406\.03505 \(2024\)