Mutual information and sensitivity analysis for feature selection in customer targeting: a comparative study

# Mutual information and sensitivity analysis for feature selection in customer targeting: a comparative study
Source: [https://arxiv.org/html/2608.20447](https://arxiv.org/html/2608.20447)
1Universidad Nacional de Tres de Febrero, Caseros, Argentina2Instituto Universitário de Lisboa \(ISCTE\-IUL\), ISTAR\-IUL, Lisboa, PortugalALGORITMI Research Centre, University of Minho, Guimarães, Portugal3Dataxplore, Trenque Lauquen, Argentina4Boldt Gaming, Buenos Aires, Argentina

###### Abstract

Feature selection is a highly relevant task in a data\-driven knowledge discovery project\. Several techniques have been developed aiming at finding the features that influence most an outcome to predict, including mutual information and, in recent years, the data\-based sensitivity analysis\. The present research focus on analyzing the advantages and disadvantages of each of these two techniques, by applying both to a bank telemarketing case\. Thereafter, a logistic regression model is built on the tuned set of features identified by each of the two techniques as the most influencing set of features on the success of a telemarketing contact, in a total of 13 features for mutual information and 9 features for the data\-based sensitivity analysis\. The latter performs better for lower values of false positives while the former is slightly better for a higher false positive ratio\. Thus, mutual information becomes a better choice if bank managers intend to reduce slightly the cost of contacts without risking losing a high number of successes\. Such results show that mutual information, although not recent, is still a valid method for feature selection\. On the other side, the data\-based sensitivity analysis’ selection achieved good prediction results with less features\.

Keywords: Feature selection; Mutual information; Sensitivity analysis; Customer targeting; Direct marketing; Modeling\.

## 1Introduction

Customer targeting \(CT\) is a classical problem addressed by Business Intelligence \(BI\) methods and techniques\. It involves finding the right customers who to target at within the context of a marketing campaign for selling the campaign product or service\[[40](https://arxiv.org/html/2608.20447#bib.bib1)\]\. Typical cutting edge approaches include using data mining \(DM\) for unveiling the potential knowledge through patterns of information hidden in big data repositories\[[43](https://arxiv.org/html/2608.20447#bib.bib2)\]\. DM adopts the best practices inherited both from classical statistics and artificial intelligence, in an attempt to take advantage from both to enhance knowledge extraction from raw data\[[12](https://arxiv.org/html/2608.20447#bib.bib3)\]\.

In recent years, industries worldwide have experienced peaks and troughs of enthusiasm arisen from the high expectations of benefiting from novel technologies and approaches introduced by DM\[[10](https://arxiv.org/html/2608.20447#bib.bib4)\]\. Discovering the best customers for targeting at a specific moment in time has proven to be NP\-hard\[[30](https://arxiv.org/html/2608.20447#bib.bib5)\]\. In real world, a vast number of characteristics and contextual specificities may potentially affect customer’s receptivity for acquiring a product\. While recent technologies and DM procedures have progressively been increasing their capabilities of analyzing large quantities of data, there is a growing need derived from data availability to identify which are the features that may potentially influence an outcome and which are the ones that are irrelevant and should be discarded, for these may directly contribute for misleading DM algorithms\[[26](https://arxiv.org/html/2608.20447#bib.bib7)\]\. Also, the larger the number of features, the slower and more complex is the execution of the DM algorithm in pursuit for the best possible solution, given the exponential growth of possibilities that the algorithm needs to explore\[[44](https://arxiv.org/html/2608.20447#bib.bib8)\]\. Hence, feature selection is a highly relevant task in any DM approach, constituting a key step where a large portion of the global effort should be spent\[[8](https://arxiv.org/html/2608.20447#bib.bib6)\]\.

Several techniques have been introduced and applied for feature selection\. In\[[2](https://arxiv.org/html/2608.20447#bib.bib9)\]the authors conducted a survey, identifying three main methods, filter, wrapper, and embedded, while also mentioning the application of other techniques such as using unsupervised learning and ensemble methods\. Table[1](https://arxiv.org/html/2608.20447#S1.T1)summarizes their categorization\.

MethodDescriptionExamples of techniquesFiltervariable ranking techniques as the principle criteria for variable selection by orderingCorrelation criteriaMutual informationWrapperuse the predictor as a black box and the predictor performance as the objective function to evaluate the variable subsetSequential selection algorithmsHeuristic search algorithmsSensitivity analysisEmbeddedreduce the computation time taken up for reclassifying different subsets which is done in wrapper methods by incorporating the feature selection as part of the training processSVM\-RFE \(Recursive Feature Elimination\)Othersseveral techniques that do not fit in the remaining three methodsClusteringEnsembleSensitivity analysis

Table 1:Feature selection methods \(adapted from\[[2](https://arxiv.org/html/2608.20447#bib.bib9)\]\)Mutual information \(MI\) is one of the most widely adopted feature selection techniques, with the earliest studies dating back to the nineteen nineties\[[1](https://arxiv.org/html/2608.20447#bib.bib10)\]\. The concept underlying MI is to measure the mutual dependence between two random features by identifying how much information of one of the features can be obtained from the other feature\. Thus, it is linked to the entropy of a random feature, given by the amount of information held in the feature\[[31](https://arxiv.org/html/2608.20447#bib.bib11)\]\.

The usage of sensitivity analysis \(SA\) for feature selection in DM projects has been studied at least since the dawn of the new millennium\[[9](https://arxiv.org/html/2608.20447#bib.bib19)\]\. The main idea behind SA is to assess model’s sensitivity to the variation of each of the input features on the predicted outcome: the more sensitive is the model, the more relevant is the effect of changing the input feature on the outcome\. In this context, SA may be considered a wrapper method, according to categorization identified in Table[1](https://arxiv.org/html/2608.20447#S1.T1), even though SA may also be included within the model training process \(thus, in this latter case, it would become an embedded method\)\.

Most DM projects need to include a data preparation step, where usually occurs a feature selection procedure\[[14](https://arxiv.org/html/2608.20447#bib.bib13)\]\. Being CT a typical problem addressed through DM, makes of it an ideal candidate for the application of feature selection methods\. Thus, several studies were published related to the application of feature selection to CT\[[37](https://arxiv.org/html/2608.20447#bib.bib14),[25](https://arxiv.org/html/2608.20447#bib.bib15)\]\. A recent study authored by\[[38](https://arxiv.org/html/2608.20447#bib.bib16)\]verses on the impact of feature selection in direct marketing\. Their work analyzed three filter methods for feature selection \(correlation\-based feature selection, subset consistency, and symmetrical uncertainty\), concluding that symmetrical uncertainty resulted in better models, outperforming others using the remaining two methods studied and a model without any feature selection procedure\.

While there are several studies published on feature selection using MI, and a few using SA, none performed a direct comparison on both methods to assess the pros and cons on using each\. Furthermore, even though a handful of recent studies were found comparing feature selection methods through practical applications\[[38](https://arxiv.org/html/2608.20447#bib.bib16)\], none considered SA\. The main contributions of this paper are as follows:

- ∙\\bulletComparing mutual information with sensitivity analysis for feature selection, by testing both methods on a real case CT problem;
- ∙\\bulletAssessing the advantages and disadvantages of adopting each of the methods for feature selection, by cross\-validating the results achieved on the experiments with real data with the background provided by the literature on the subject;
- ∙\\bulletDrawing the insights on each method that may lead scholars and researchers on the adoption of each for a wide range of data\-driven approaches to address real\-world problems\.

This paper is organized as follows\. Section[2](https://arxiv.org/html/2608.20447#S2)presents a summary on the literature for MI, SA and feature selection applied to CT\. In Section[3](https://arxiv.org/html/2608.20447#S3), the materials and methods adopted for the experiments are described\. Results and evaluation of applying both methods are discussed in Section[4](https://arxiv.org/html/2608.20447#S4)\. Finally, conclusions are drawn in the last section\.

## 2Background

### 2\.1Mutual information

Entropy and mutual information \(MI\) are well known concepts in Communications and Information Theory\. They were originally introduced by Claude Shannon in a seminal paper\[[36](https://arxiv.org/html/2608.20447#bib.bib38)\], in order to find the optimal coding of a source on one hand and a noisy channel on the other\. Entropy is related to uncertainty or information content of a random variable\. From this point of view, an eventiihaving probability of occurrencepip\_\{i\}has an information content of:

I=−log⁡piI=\-\\log p\_\{i\}\(1\)
The base of logarithm defines the unit, base two logarithm gives units in bits\. When more probable is the event, less information gives its occurrence\. The expectation of \([1](https://arxiv.org/html/2608.20447#S2.E1)\) gives the average information content of such set of events:

H\(X\)=−∑pilogpiH\(X\)=\-\\sum p\_\{i\}\\log p\_\{i\}\(2\)The expression \([2](https://arxiv.org/html/2608.20447#S2.E2)\) is the entropy of the random variableXXin such a way that the eventiicorresponds to the valuexix\_\{i\}, i\.e\.p⁡\(xi\)=pip\(x\_\{i\}\)=p\_\{i\}\.

Entropy is bounded by the cardinality of the set of outcomes:H⁡\(X\)≤log⁡\|𝒳\|H\(X\)\\leq\\log\|\\mathcal\{X\}\|and attains its maximum when the events follow a uniform distributionpi=1\|𝒳\|p\_\{i\}=\\frac\{1\}\{\|\\mathcal\{X\}\|\}\. The bigger the entropy, the more random are the events, then, the occurrence of an event gives more information, though they are less predictable\.

Considering two random variables with a given joint probabilityp⁡\(X,Y\)p\(X,Y\), the joint entropy is defined as:

H\(X,Y\)=−∑x∈𝒳,y∈𝒴p\(x,y\)logp\(x,y\)H\(X,Y\)=\-\\sum\_\{x\\in\\mathcal\{X\},y\\in\\mathcal\{Y\}\}p\(x,y\)\\log p\(x,y\)\(3\)
And the conditional entropy is defined as:

H\(X\|Y\)=−∑x∈𝒳,y∈𝒴p\(x,y\)logp\(x\|y\)H\(X\|Y\)=\-\\sum\_\{x\\in\\mathcal\{X\},y\\in\\mathcal\{Y\}\}p\(x,y\)\\log p\(x\|y\)\(4\)
Mutual information between two random variables is defined as follows:

I⁡\(X,Y\)=∑x∈𝒳,y∈𝒴p⁡\(x,y\)​log⁡p⁡\(x,y\)p⁡\(x\)​p​\(y\)I\(X;Y\)=\\sum\_\{x\\in\\mathcal\{X\},y\\in\\mathcal\{Y\}\}p\(x,y\)\\log\\frac\{p\(x,y\)\}\{p\(x\)\\;p\(y\)\}\(5\)
From \([5](https://arxiv.org/html/2608.20447#S2.E5)\) we can derive a relation between mutual information and entropy:

I⁡\(X,Y\)\\displaystyle I\(X;Y\)=\\displaystyle=H⁡\(X\)−H⁡\(X\|Y\)\\displaystyle H\(X\)\-H\(X\|Y\)\(6\)=\\displaystyle=H⁡\(Y\)−H⁡\(Y\|X\)\\displaystyle H\(Y\)\-H\(Y\|X\)
Mutual information definition can be extended to sets of random variablesXn=\{X1,X2,⋯,Xn\}X^\{n\}=\\\{X\_\{1\},X\_\{2\},\\cdots,X\_\{n\}\\\}andYn=\{Y1,Y2,⋯,Yn\}Y^\{n\}=\\\{Y\_\{1\},Y\_\{2\},\\cdots,Y\_\{n\}\\\}:

I⁡\(Xn,Yn\)=∑xn∈𝒳n,yn∈𝒴np⁡\(xn,yn\)​log⁡p⁡\(xn,yn\)p⁡\(xn\)​p​\(yn\)I\(X^\{n\};Y^\{n\}\)=\\sum\_\{x^\{n\}\\in\\mathcal\{X\}^\{n\},y^\{n\}\\in\\mathcal\{Y\}^\{n\}\}p\(x^\{n\},y^\{n\}\)\\log\\frac\{p\(x^\{n\},y^\{n\}\)\}\{p\(x^\{n\}\)\\;p\(y^\{n\}\)\}\(7\)where𝒳m\\mathcal\{X\}^\{m\}and𝒴m\\mathcal\{Y\}^\{m\}are the set of outcomes ofxnx^\{n\}andymy^\{m\}\.

A situation where redundant information can be removed arises when variables are connected in a Markov chain, as shown in Figure[1](https://arxiv.org/html/2608.20447#S2.F1)\. A well known relation for this case is given by the data processing inequalityI⁡\(X1,X2\)≥I⁡\(X1,Y\)I\(X\_\{1\};X\_\{2\}\)\\geq I\(X\_\{1\};Y\), an alternative inequality is demonstrated in Lemma[1](https://arxiv.org/html/2608.20447#Thmtheorem1)in a similar way\.

Figure 1:Connection of variables for lemma[1](https://arxiv.org/html/2608.20447#Thmtheorem1)###### Lemma 1\.

If the random variablesX1X\_\{1\},X2X\_\{2\},YYare connected in a Markov chainX1→X2→YX\_\{1\}\\rightarrow X\_\{2\}\\rightarrow Y, then:

1. 1\.I⁡\(X1,X2,Y\)=I⁡\(X2,Y\)I\(X\_\{1\},X\_\{2\};Y\)=I\(X\_\{2\};Y\)
2. 2\.I⁡\(X2,Y\)≥I⁡\(X1,Y\)I\(X\_\{2\};Y\)\\geq I\(X\_\{1\};Y\)

###### Proof\.

Applying twice the chain rule to the mutual information:

I⁡\(X1,X2,Y\)\\displaystyle I\(X\_\{1\},X\_\{2\};Y\)=\\displaystyle=I⁡\(X1;Y\|X2\)\+I⁡\(X2,Y\)\\displaystyle I\(X\_\{1\};Y\|X\_\{2\}\)\+I\(X\_\{2\};Y\)\(8\)=\\displaystyle=I⁡\(X2;Y\|X1\)\+I⁡\(X1,Y\)\\displaystyle I\(X\_\{2\};Y\|X\_\{1\}\)\+I\(X\_\{1\};Y\)\(9\)
By the Markov property:

I⁡\(X1;Y\|X2\)=0I\(X\_\{1\};Y\|X\_\{2\}\)=0\(10\)
Replacing \([10](https://arxiv.org/html/2608.20447#S2.E10)\) in \([8](https://arxiv.org/html/2608.20447#S2.E8)\) we prove the first part\.

Taking into account that the mutual information is always greater than 0 \(see for example\[[7](https://arxiv.org/html/2608.20447#bib.bib39)\]\), from \([8](https://arxiv.org/html/2608.20447#S2.E8)\), \([9](https://arxiv.org/html/2608.20447#S2.E9)\) and \([10](https://arxiv.org/html/2608.20447#S2.E10)\) we getI⁡\(X2,Y\)≥I⁡\(X1,Y\)I\(X\_\{2\};Y\)\\geq I\(X\_\{1\};Y\), and the lemma is proved\.

∎

Then, we can point out that these functions from the information theory are quite suitable to eliminate redundant information, not generally taken into account with other methods\.

A communication channel is a device or medium capable of transmitting information\. The input information is carried out to the output\. Since any mechanism of transmitting information is not perfect, some noise is introduced in the process\. Then, the input and output information are not the same but related\. We can use the mutual information as a measure of that relation\. We will consider thereafter thatXXis the random variable at the input andYYis the output random variable\.

According to the source coding theorem, we know that the entropy is a measure of the average bits of information necessary to code the outcomes of a given random variable\. In this way,H⁡\(X\)H\(X\)is a measure of the input information to the channel,H⁡\(Y\)H\(Y\)the information content at the output,I⁡\(X,Y\)I\(X;Y\)the transmitted information, and taking into account the relations \([6](https://arxiv.org/html/2608.20447#S2.E6)\):H⁡\(Y\|X\)H\(Y\|X\)is a measure of the noise introduced by the channel\. The conditional entropyH⁡\(X\|Y\)H\(X\|Y\)is called equivocation or ambiguity and it must be subtracted to the input information in order to obtain the transmitted information\. According to the channel coding theorem, the Mutual Information gives the channel capacity and determines the maximum rate of information transmitted by the channel, see details in\[[36](https://arxiv.org/html/2608.20447#bib.bib38)\]and\[[7](https://arxiv.org/html/2608.20447#bib.bib39)\]\.

Mutual Information based feature selection consists in choosing the set of variables raising the most of the information of the output variable following a given criterion\. The process starts by adding features from those carrying the most of the information until a stopping criterion is reached\. Since the mutual information between the output variableYYand a given subset of input variablesXmX^\{m\}is given by:

I⁡\(Xm,Y\)=H⁡\(Y\)−H⁡\(Y\|Xm\)I\(X^\{m\};Y\)=H\(Y\)\-H\(Y\|X^\{m\}\)\(11\)the estimation ofI⁡\(Xm,Y\)I\(X^\{m\};Y\)involves the estimation ofP⁡\(Y\|Xm\)P\(Y\|X^\{m\}\)\. Then, the bigger the setXmX^\{m\}, the less reliable is the estimation ofP⁡\(Y\|Xm\)P\(Y\|X^\{m\}\)\. Therefore, the maximum of the product of information gain with reliability of estimation is used as the stopping rule, as indicated in the flow chart shown in fig\.[2](https://arxiv.org/html/2608.20447#S2.F2)\. Other stopping rules for variables addition were considered in\[[13](https://arxiv.org/html/2608.20447#bib.bib41)\]\. Note that the relative information gainI​Gm=I⁡\(Xm,Y\)H⁡\(Y\)IG^\{m\}=\\frac\{I\(X^\{m\};Y\)\}\{H\(Y\)\}goes from00whenXmX^\{m\}andYYare independent andH⁡\(Y\|Xm\)=H⁡\(Y\)H\(Y\|X^\{m\}\)=H\(Y\)to 100% whenYYis a deterministic function ofXmX^\{m\}andH⁡\(Y\|Xm\)=0H\(Y\|X^\{m\}\)=0\.

Figure 2:MI feature selection methodAs a result, MI theory may help to cope with the task of feature selection by identifying the subset of features that minimizesH⁡\(Y\|X\)H\(Y\|X\), i\.e\., that maximizes the information underlying in the dataset,I⁡\(X,Y\)I\(X;Y\)\. Thus, features introducing further entropy in the communication channel may be discarded, helping to guide DM algorithms in building a model that understands the intrinsic relationships underneath the original data\.

### 2\.2Sensitivity analysis

The usage of sensitivity analysis \(SA\) for providing insights on complex models dates back from the nineteen nineties, with a special emphasis on climate and environmental models\[[16](https://arxiv.org/html/2608.20447#bib.bib17)\]\. However, the advent of DM has introduced highly complex models with intrinsic convoluted relations that are hardly disentangled\. Some of the most widely used of those models include machine learning algorithms such as neural networks in various formats and versions, and support vector machines\. SA has been proposed and analyzed in the literature for input feature evaluation from data mining models\. In Ref\.\[[20](https://arxiv.org/html/2608.20447#bib.bib18)\]a computationally efficient one\-dimensional method is presented, by varying one input at a time through its possible range of values and keeping the remaining input features constant\. Subsequent studies explored further SA as a means for understanding the impact each of the features that contributed to a model implementation had on the predicted outcome\[[9](https://arxiv.org/html/2608.20447#bib.bib19),[22](https://arxiv.org/html/2608.20447#bib.bib20)\]\. By providing a procedure to evaluate the relevance of input features from models, several studies have included SA as a method for feature selection, hence choosing the most relevant features and discarding the least relevant\[[24](https://arxiv.org/html/2608.20447#bib.bib21)\]\. Although SA requires that a model is previously available for assessing feature relevance since it focus solely on the features, it can virtually be applied to any type of predictive model\. Recent developments resulted in novel SA techniques such as the data\-based SA \(DSA\), introduced by\[[5](https://arxiv.org/html/2608.20447#bib.bib12)\]in 2013\. This procedure uses random samples from the data used to train the model for assessing feature relevance by changing the input features simultaneously, thus considering the relations between features \(Figure[3](https://arxiv.org/html/2608.20447#S2.F3)\)\.

![Refer to caption](https://arxiv.org/html/2608.20447v1/dsa.png)Figure 3:Data\-based sensitivity analysisDSA has been applied since then in a large spectrum of domains and problems, such as bank marketing\[[27](https://arxiv.org/html/2608.20447#bib.bib22)\], wine quality assessment\[[4](https://arxiv.org/html/2608.20447#bib.bib23)\], jet grouting formulations\[[39](https://arxiv.org/html/2608.20447#bib.bib24)\]and social media performance metrics\[[28](https://arxiv.org/html/2608.20447#bib.bib25)\]\. However, the only study using DSA specifically for feature selection is the work by Moro, Cortez and Rita, which applied such technique in a bank telemarketing case and resulted in a thread of published articles\[[26](https://arxiv.org/html/2608.20447#bib.bib7),[27](https://arxiv.org/html/2608.20447#bib.bib22),[25](https://arxiv.org/html/2608.20447#bib.bib15)\]\. Their work adopted DSA but lacked in assessing the advantages and disadvantages of sensitivity analysis when compared to other feature selection methods\. The present paper represents the first attempt in filling such gap\.

### 2\.3Feature selection in customer targeting

Customer targeting \(CT\) is the marketing procedure of optimizing the selection of customers who to target within the context of a marketing campaign to meet campaign goals, usually, the acquisition of a product or service\[[3](https://arxiv.org/html/2608.20447#bib.bib26)\]\. CT can be viewed as a branch of an integrated Customer Relationship Management \(CRM\) strategy with a focus on building customer equity\[[34](https://arxiv.org/html/2608.20447#bib.bib27)\]\. Other terms that are directly related to CT include direct marketing and database marketing, with the former being almost a synonymous\[[21](https://arxiv.org/html/2608.20447#bib.bib28)\], while the latter can be also associated with the need for a customer database to support CRM strategies\[[34](https://arxiv.org/html/2608.20447#bib.bib27)\]\. It should be noted that the task of discovering the most likely positive responders to customer targeting has been proven to be NP\-hard\[[30](https://arxiv.org/html/2608.20447#bib.bib5)\]\. CT provides an interesting ground for testing predictive machine learning techniques, with a large number of published studies alleging the discovery of predictive knowledge that may be used to benefit the success of CT\[[21](https://arxiv.org/html/2608.20447#bib.bib28),[29](https://arxiv.org/html/2608.20447#bib.bib29)\]\. Nevertheless, few of those works have seen a real production environment, effectively leveraging business\[[32](https://arxiv.org/html/2608.20447#bib.bib30)\]\.

Feature selection is a key task in every DM projects\[[8](https://arxiv.org/html/2608.20447#bib.bib6)\]\. The main goal is to find the minimum set of features that optimize results translated in terms of model accuracy in fitting new data for the problem being addressed\[[43](https://arxiv.org/html/2608.20447#bib.bib2)\]\. Also, by reducing the number of features used for modeling, the procedure for training the model becomes lesser computationally expensive, making it feasible to be executed on a daily or more frequent basis, for incorporating the subtle changes derived from immediate previous contacts\[[23](https://arxiv.org/html/2608.20447#bib.bib35)\]\. For example, a bad news on the company or product widely spread through social media may directly affect the subsequent contacts\[[11](https://arxiv.org/html/2608.20447#bib.bib36)\]\. Thus, model retraining for learning with new occurrences needs to occur often\. One option is to use a rolling windows procedure where the window of data for training the model slides for keeping pace with time, an approach that may be adopted for several time evolving problems such as stock markets\[[35](https://arxiv.org/html/2608.20447#bib.bib37)\]and telemarketing\[[26](https://arxiv.org/html/2608.20447#bib.bib7)\]\. The longer the algorithm takes to run the modeling procedure, the more likely the model does not adapt quickly enough to new information\. Hence, selecting the right amount of features is in demand for problems with constant shifts in the influence features have on the outcome\.

Feature selection using MI has been a subject of research in numerous problems\. Moreover, studies are usually devoted to testing new feature selection approaches to well\-known datasets, not focusing explicitly on the advantages to the business associated with the problems being addressed\[[17](https://arxiv.org/html/2608.20447#bib.bib31),[15](https://arxiv.org/html/2608.20447#bib.bib32)\]\. However, no studies were found on feature selection using MI specifically focusing on CT, only a few papers published related to customer churning\[[18](https://arxiv.org/html/2608.20447#bib.bib33),[42](https://arxiv.org/html/2608.20447#bib.bib34)\]\. DSA application to CT for feature selection has been the subject of study of Moro et al\., as stated in Section[2\.2](https://arxiv.org/html/2608.20447#S2.SS2)\. The present study is focused in filling such void while at the same time performing a novel comparison between both methods, MI and DSA\.

## 3Materials and Methods

### 3\.1Real case

Bank telemarketing is a specific case of direct marketing where the customers of a bank are contacted and offered products or services through phone calls, although other direct channels such as email may be used\[[26](https://arxiv.org/html/2608.20447#bib.bib7)\]\. For the experiments presented in this paper, the dataset published in the University of California Machine Learning Repository \(http://archive\.ics\.uci\.edu/ ml/\) was adopted\. Such dataset was studied by numerous scholars and researchers, as the high number of page hits shows, above two hundred thousand\. As a result, several studies have been published using its data, with the most for assessing machine learning and DM algorithms’ capabilities\[[42](https://arxiv.org/html/2608.20447#bib.bib34)\], and a few for feature selection\[[41](https://arxiv.org/html/2608.20447#bib.bib40)\]\. This dataset encompasses a total of 41,188 phone contacts conducted by human agents from a Portuguese bank between 2008 and 2010, with the goal of selling an attractive long\-term deposit, in an attempt of retaining customers’ financial assets in the institution\. It should be stressed that all contacts are real, implying that it represents a real problem and to which feature selection may provide interesting benefits in reducing the features needed for modeling the outcome, choosing only influencing features while at the same time reducing model retraining duration\. Each contact is characterized by twenty features, with some related to personal customer data \(e\.g\., age\), others to the contact itself \(e\.g\., call duration\) and previous calls made within the context of older campaigns \(e\.g\., the outcome of previous contact\), and the remaining related to the social and economic context that characterizes the country \(e\.g\., number of employed people\)\. Table[2](https://arxiv.org/html/2608.20447#S3.T2)describes the list of features\. More details can be obtained from Ref\.\[[26](https://arxiv.org/html/2608.20447#bib.bib7)\]\. The target outcome is the 21stfeature from the dataset, concealing a binary value \(yes/no\) which represents the contact result: “yes” if the customer subscribed the deposit; “no” otherwise\.

FeatureType and descriptionGroupagenumericCustomerjobtype of job \(categorical \- 12 possible values\)maritalmarital status \(categorical \- 4 possible values\)education\(categorical \- 8 possible values\)defaulthas credit in default? \(categorical: “no”,“yes”,“unknown”\)housinghas housing loan? \(categorical: “no”,“yes”,“unknown”\)loanhas personal loan? \(categorical: “no”,“yes”,“unknown”\)contactcontact communication type \(categorical: “cellular”,“telephone”\)Contactmonthlast contact month of year \(categorical\)day\_of\_weeklast contact day of the week \(categorical\)durationlast contact duration, in seconds \(numeric\)campaignnumber of contacts performed during this campaignOtherpdaysnumber of days that passed by after the client was last contacted from a previous campaignpreviousnumber of contacts performed before this campaign and for this clientpoutcomeoutcome of the previous marketing campaignemp\.var\.rateemployment variation rate \- quarterly indicator \(numeric\)Contextcons\.price\.idxconsumer price index \- monthly indicator \(numeric\)cons\.conf\.idxconsumer confidence index \- monthly indicator \(numeric\)euribor3meuribor 3 month rate \- daily indicator \(numeric\)nr\.employednumber of employees \- quarterly indicator \(numeric\)

Table 2:List of input features \(from https://archive\.ics\.uci\.edu/ml/datasets/Bank\+Marketing\)
### 3\.2Experimental procedure

The bank telemarketing dataset was first assessed in terms of feature relevance by both methods studied, MI and DSA\. Each method has its own specificities and procedures, as mentioned in sections[2\.1](https://arxiv.org/html/2608.20447#S2.SS1)and[2\.2](https://arxiv.org/html/2608.20447#S2.SS2)\. MI evaluates the amount of information concealed in each of the features, whereas DSA assesses the model in terms of the influence on the outcome by changing the input features\. The experimental setup for the case of MI is solely the dataset with the data, as detailed in Section[2\.1](https://arxiv.org/html/2608.20447#S2.SS1), while the DSA required that a model was previously built for assessing feature relevance in terms of the sensitivity of the model to changes on input features, as shown in Figure[3](https://arxiv.org/html/2608.20447#S2.F3)\. Therefore, while MI selects the features according to the information each of them contains when compared to the remaining, DSA ranks features in terms of relevance for the model\. For the latter, all the features that did not encompass individually at least 2% of relevance were discarded\. Figure[4](https://arxiv.org/html/2608.20447#S3.F4)summarizes the approach followed\.

![Refer to caption](https://arxiv.org/html/2608.20447v1/procedure.png)Figure 4:ProcedureIn order to simulate a sliding window, the dataset was divided successively in different training and testing sets using a ten fold cross\-validation procedure\. The testing set is composed by the contacts whose willingness of acquiring the product constitutes the outcome to predict\[[33](https://arxiv.org/html/2608.20447#bib.bib44)\]\. This set is chosen by a window that takes 10% of the total records, starts at the first record and shifts to the next 10% of records without overlapping\. In each experiment, the training set is composed by the other 90% records\. At each fold, from the training set, the features were selected and used for building a predictive model which is applied to the testing set, in a procedure similar to Ref\.\[[26](https://arxiv.org/html/2608.20447#bib.bib7)\]\. At the end of the 10 fold experiments, a score of the probability of acquiring the offered product was computed for each contact\. That score was then used to build the ROC curve and find the confusion matrices shown below for different cutoff probabilities\. The selected features are then used for implementing a simple logistic regression model to fit data for predicting the outcome on the contacts\. The usage of logistic regression provides a direct means for measuring how modeling with the selected feature behaves in both cases, for allowing a direct comparison\. While more complex machine learning techniques could be used \(e\.g\., neural networks or support vector machines\), the goal of the present study is to facilitate a comparison of both feature selection procedures, not putting emphasis on the modeling scenarios, where other studies have already focused on the analyzed dataset\. Also, to keep coherence in all experiments, the logistic regression was also chosen for extracting feature relevance during feature selection from DSA\. Additionally, since DSA is based on ranking feature relevance, for computing the model, all top ranked features with a summed relevance of at least 90% were included, discarding the remaining\. Finally, the prediction results are analyzed in the light of comparing both methods using both the receiver operating characteristic curve \(ROC\) and confusion matrices\. Also, computational performance is evaluated, for a lighter method in terms of execution may allow a global frequently run learning procedure to be scheduled more often, as stated in Section[2\.3](https://arxiv.org/html/2608.20447#S2.SS3)\.

## 4Results and Discussion

As remarked in\[[26](https://arxiv.org/html/2608.20447#bib.bib7)\], a predictive model does not hold any knowledge on future occurrences of the problem; thus, the dataset should be striped off of any feature only known after contact execution, such as the call duration\. Furthermore, in a real predictive system, a campaign is launched without knowing when will the calls be made, as these depend on both agent and especially client availability\. Therefore, for the experimental setup, all features related to the current campaign were removed, namely: ”contact”, ”month”, ”day” and ”duration”\. The only exception was ”campaign”, which deserved a detailed analysis: this field indicates the number of contacts performed during this campaign, i\.e\., how many times was the contact rescheduled, which may happen due to several reasons, such as a machine answered the call and the agent decided to reschedule it, or the client asked to be recalled later\. By taking into account that a telemarketing campaign can last a year, one may consider a dynamical model that take this field into account, incorporating multiple calls to the same client within the same campaign; therefore, this feature was included in the present analysis\.

Since the software powerhouse111http://www\.dataxplore\.com\.ar/tecnologia\.php\#Powerhouseperforms a segmentation process based on information theory, such product was chosen for obtaining the metrics\. The software calculates the mutual information between each attribute and the output variable separately by estimating the joint probability, then, it chooses the set of variables carrying most of the information according to the criterion explained in sec\.[2\.1](https://arxiv.org/html/2608.20447#S2.SS1)\. The set of the selected variables are shown in table[3](https://arxiv.org/html/2608.20447#S4.T3)\.

Table 3:Selected variables by the MI method, NF stands for the number of folds the attribute has been taken into account because of its relevance\.Elimination of redundant information occurs in the present data with the variable ”conf\.price\.idx”\. This variable can be totally predicted by ”cons\.conf\.idx” and ”nr\.employed” as shown in table[4](https://arxiv.org/html/2608.20447#S4.T4)\.

Table 4:Information content and prediction of the variable ”conf\.price\.idx”\.As a result, we can say that the variables mentioned before are connected as shown in Fig\.[5](https://arxiv.org/html/2608.20447#S4.F5), in a similar way as it was presented in lemma[1](https://arxiv.org/html/2608.20447#Thmtheorem1)\.

Figure 5:Prediction of variable ”conf\.price\.idx”Similar information content tables are shown for the other discarded variables ”default” and ”previous”\.

Table 5:Information content and prediction of the variable ”previous”\.Table 6:Information content and prediction of the variable ”default”\.For the case of DSA feature selection, the R statistical tool was chosen \(https://cran\.r\-project\.org/\)\. R is an open source framework focusing on data analysis problems, allowing contributions with independent packages developed by numerous researchers worldwide\[[19](https://arxiv.org/html/2608.20447#bib.bib42)\]\. Additionally, the rminer package implements the DSA algorithm\[[6](https://arxiv.org/html/2608.20447#bib.bib43)\]; therefore, it was adopted for all experiments related to DSA\. Selected variables by DSA method are shown in table[7](https://arxiv.org/html/2608.20447#S4.T7)\. As previously explained, it should be noted that within each fold of execution, all features summing up to 90% of global relevance were included\. Interestingly, these features were considered for each of the ten folds, emphasizing its relevance for the problem and data being addressed\.

Table 7:Selected variables by the DSA method\.Once a set of highly relevant features are selected based on the information content for MI, or based on model sensitivity to such features for DSA \(Table[8](https://arxiv.org/html/2608.20447#S4.T8)\), a predictive model may be built using any available DM technique, from naive Bayes to the most recent support vector machines, neural networks and genetic algorithms \(e\.g\.,\[[26](https://arxiv.org/html/2608.20447#bib.bib7)\]\)\. Considering the focus of this study is feature selection, a simple logistic regression method was chosen for predicting contact outcome, for assessing the efficiency and accuracy of the features selected with both methods\. This is the simplest algorithm of those analyzed by\[[26](https://arxiv.org/html/2608.20447#bib.bib7)\]\. The ROC curves for both models are drawn in Figure[6](https://arxiv.org/html/2608.20447#S4.F6)\. For further understanding the effects of using each predictive model built on each set of features, four confusion matrices are computed\. On Tables[10](https://arxiv.org/html/2608.20447#S4.T10)and[10](https://arxiv.org/html/2608.20447#S4.T10), results are extracted considering a typical cutoff probability 0\.5, i\.e\., in which the most likely outcome is considered a success if the model predicts it with 50% or more of probability, whereas on Tables[12](https://arxiv.org/html/2608.20447#S4.T12)and[12](https://arxiv.org/html/2608.20447#S4.T12)the cutoff is lowered to just 10%, to account for the fact that this particular bank intends to increase efficiency with a especial emphasis on avoiding loosing successful contacts, considering lost deposit subscriptions directly implicates on missing business opportunities for retaining important financial assets in a crisis period \(thus, the cost of loosing a successful contact is much higher than the gain of avoiding an unuseful unsuccessful contact\)\[[26](https://arxiv.org/html/2608.20447#bib.bib7)\]\.

Table 8:Selected features for both methods\.Figure 6:ROC curves\.Table 9:Confusion Matrix for the MI method for cutoff probability 50%\.
Table 10:Confusion Matrix for the DSA method for cutoff probability 50%\.

Table 11:Confusion Matrix for the MI method for cutoff probability 10%\.
Table 12:Confusion Matrix for the DSA method for cutoff probability 10%\.

It is interesting to observe from the confusion matrices that an increasing number of false positives \(FP\) turns MI slightly better in predicting contact outcome than DSA, whereas DSA is clearly better for smaller FPs\. Nevertheless, as stated previously, the results of MI maybe preferable if accounted that the cost of making a call is far less than the benefits of hitting a customer willing of getting the product\. Another advantage of MI method is processing time, as results are obtained faster, since no heavy CPU consume is involved for modeling as it is the case of DSA\. For the experiments conducted, MI procedure took just a few seconds, whereas DSA took around half a minute in an IntelTMI3 processor\. This is a highly relevant benefit if more complex model techniques are used, as DSA depends on a model being built first\. Furthermore, such advantage may be particularly emphasized for larger datasets\.

Comparing tables[7](https://arxiv.org/html/2608.20447#S4.T7)and[3](https://arxiv.org/html/2608.20447#S4.T3)several important remarks may be stated:

1. 1\.Both feature selection methods clearly achieve different results\.
2. 2\.The most relevant features for one method is considered as little relevant for the other\. Such is the case of ”emp\.var\.rate” and ”cons\.conf\.idx”\.
3. 3\.Contrarily to MI, the number of features selected by DSA remains the same along the 10\-fold experiments\.
4. 4\.The set of features selected by MI is bigger than that selected by DSA\.

The previous remarks lead to an interesting analysis\. It seems to be that a given estimate like DSA does not reveal the inherent dependency among strongly correlated features\. This situation can occur if the evaluation estimate considers a value where the independent variable does not lead to big variations in the estimate, like in a possible flat portion in the regression curve\. This may be observed from Tables[7](https://arxiv.org/html/2608.20447#S4.T7)and[3](https://arxiv.org/html/2608.20447#S4.T3), where the features selected by DSA remain the same through the folds, contrarily to the MI method\. Also, from the MI method, a feature can be discarded if there is not enough values as to get a good level of confidence, explaining why many variables were discarded in some folds, as shown in Table[3](https://arxiv.org/html/2608.20447#S4.T3), column NF\.

It is possible to observe that MI has not taken into account the variables ”conf\.price\.idx”, ”default” and ”previous”; such finding has risen the interest in analyzing how each of these variables are related to the remaining selected by MI\. This analysis can be made from Tables[4](https://arxiv.org/html/2608.20447#S4.T4),[5](https://arxiv.org/html/2608.20447#S4.T5)and[6](https://arxiv.org/html/2608.20447#S4.T6)\. Table[4](https://arxiv.org/html/2608.20447#S4.T4)shows that ”conf\.price\.idx” can be totally predicted by ´´cons\.conf\.idx´´ and ´´nr\.employed´´\. A similar case occurs with ´´previous´´, this variable can also be almost totally predicted by other selected variables, as it can be seen from Table[5](https://arxiv.org/html/2608.20447#S4.T5)\. Also, Table[6](https://arxiv.org/html/2608.20447#S4.T6)shows that most of the information carried out by ´´default´´ is carried by other of the selected variables, thus it can be discarded\.

Another interesting result comes out from the confusion matrices shown in Tables[10](https://arxiv.org/html/2608.20447#S4.T10)and[12](https://arxiv.org/html/2608.20447#S4.T12)\. Similar behavior results for the two cutoff probability of success, it can be seen that the MI method gives confusion matrices with very good specificity and bad sensitivity\. Thus, the model hits many customers willing to get the product while at the same time failing by contacting many clients that would reject the offer\. Hence, for the empirical experiments conducted, DSA may be qualified as as more conservative, while the MI method is preferred when the cost of making a call is low and the income of selling the product is high\.

The ROC curves are similar, although the one obtained from the MI method is slightly worse given its greater number of false positives, as shown in Figure[6](https://arxiv.org/html/2608.20447#S4.F6)\.

## 5Conclusions

In this study, a comparison was conducted between two renowned feature selection methods, mutual information \(MI\) and the data\-based sensitivity analysis \(DSA\)\. Advantages and disadvantages of both methods were shown\. This important information can be used to decide the best method to use in a particular application\. For the empirical procedure, a dataset containing more than forty thousand of problem instances of the case of bank telemarketing was chosen\. In this experiment, the advantages of applying the information theory concepts in order to eliminate redundant attributes were translated in a small subset of highly relevant features which enabled modeling faster and more accurately the outcome of clients subscribing or not a deposit\. Also, the method allows getting the information content easily and rapidly, which allows that to be applied to big data sets\. Since variables carrying most of the information of the output variable are selected by the proposed method, results have shown that a simple prediction algorithm such as logistic regression can be performed with good modeling results\. On the other side, the data\-based sensitivity analysis has the disadvantage of requiring a model for extracting feature relevance\. Such drawback can halt a data mining project if the initial dataset holds a high number of features\. Nevertheless, DSA does not require to dive deeply into the model for understanding which features are lesser relevant, for it is based on assessing outcome variation by also changing input features through their range of possible values\. Using the tuned set of features obtained from each methods, in a total of 13 features for MI and 9 features for DSA from the initial 20, it is possible to observe from a logistic regression model built on each of these two sets that the receiver operating characteristic curve from DSA outperforms MI model in the lower values of false positives, while MI is slightly better for a higher false positive ratio\. Thus, if the goal of marketing managers is to reduce the number of calls made at the cost of eventually loosing some successful contacts \(true positives\), then DSA feature selection resulted better for this case; otherwise, MI’s feature selection took a small lead\. Such conclusion is highlighted in the confusion matrices obtained, with MI’s model achieving better results for predicting successes, while DSA outperforms MI on predicting failed contacts, i\.e\., when the client refused the deposit offered\. For this specific case, loosing a successful contact implicates eventual loss of the client’s financial asset, thus it is preferable to achieve a higher accuracy on predicting successes at the expense of wasting additional calls on unfruitful contacts\. Nevertheless, the results are conclusive in that MI, although a rather old method, still achieves results comparable to other more recent methods, such as DSA\.

## Acknowledgements

One of us \(NRB\) would like to thank Universidad Nacional de Tres de Febrero for financial support under grant no\. 32/15 201\.

## References

- \[1\]R\. Battiti\(1994\)Using mutual information for selecting features in supervised neural net learning\.Neural Networks, IEEE Transactions on5\(4\),pp\. 537–550\.Cited by:[§1](https://arxiv.org/html/2608.20447#S1.p4.1)\.
- \[2\]G\. Chandrashekar and F\. Sahin\(2014\)A survey on feature selection methods\.Computers & Electrical Engineering40\(1\),pp\. 16–28\.Cited by:[Table 1](https://arxiv.org/html/2608.20447#S1.T1),[§1](https://arxiv.org/html/2608.20447#S1.p3.1)\.
- \[3\]A\. M\. Cole\(2012\)Internet advertising after sorrell v\. ims health: a discussion on data privacy & the first amendment\.Cardozo Arts & Ent\. LJ30,pp\. 283–315\.Cited by:[§2\.3](https://arxiv.org/html/2608.20447#S2.SS3.p1.1)\.
- \[4\]P\. Cortez, A\. Cerdeira, F\. Almeida, T\. Matos, and J\. Reis\(2009\)Modeling wine preferences by data mining from physicochemical properties\.Decision Support Systems47\(4\),pp\. 547–553\.Cited by:[§2\.2](https://arxiv.org/html/2608.20447#S2.SS2.p2.1)\.
- \[5\]P\. Cortez and M\. J\. Embrechts\(2013\)Using sensitivity analysis and visualization techniques to open black box data mining models\.Information Sciences225,pp\. 1–17\.Cited by:[§2\.2](https://arxiv.org/html/2608.20447#S2.SS2.p1.1)\.
- \[6\]P\. Cortez\(2010\)Data mining with neural networks and support vector machines using the r/rminer tool\.InIndustrial Conference on Data Mining,pp\. 572–583\.Cited by:[§4](https://arxiv.org/html/2608.20447#S4.p6.1)\.
- \[7\]T\. M\. Cover and J\. A\. Thomas\(2006\)Elements of information theory \(wiley series in telecommunications and signal processing\)\.Wiley\-Interscience\.External Links:ISBN 0471241954Cited by:[§2\.1](https://arxiv.org/html/2608.20447#S2.SS1.p21.1.1),[§2\.1](https://arxiv.org/html/2608.20447#S2.SS1.p25.1)\.
- \[8\]P\. Domingos\(2012\)A few useful things to know about machine learning\.Communications of the ACM55\(10\),pp\. 78–87\.Cited by:[§1](https://arxiv.org/html/2608.20447#S1.p2.1),[§2\.3](https://arxiv.org/html/2608.20447#S2.SS3.p2.1)\.
- \[9\]M\. J\. Embrechts, F\. A\. Arciniegas, M\. Ozdemir, and R\. H\. Kewley\(2003\)Data mining for molecules with 2\-d neural network sensitivity analysis\.International Journal of smart engineering system design5\(4\),pp\. 225–239\.Cited by:[§1](https://arxiv.org/html/2608.20447#S1.p5.1),[§2\.2](https://arxiv.org/html/2608.20447#S2.SS2.p1.1)\.
- \[10\]A\. Gandomi and M\. Haider\(2015\)Beyond the hype: big data concepts, methods, and analytics\.International Journal of Information Management35\(2\),pp\. 137–144\.Cited by:[§1](https://arxiv.org/html/2608.20447#S1.p2.1)\.
- \[11\]J\. Goldenberg, B\. Libai, S\. Moldovan, and E\. Muller\(2007\)The npv of bad news\.International Journal of Research in Marketing24\(3\),pp\. 186–200\.Cited by:[§2\.3](https://arxiv.org/html/2608.20447#S2.SS3.p2.1)\.
- \[12\]R\. L\. Grossman, C\. Kamath, P\. Kegelmeyer, V\. Kumar, and R\. Namburu\(2013\)Data mining for scientific and engineering applications\.Vol\.2,Springer Science & Business Media\.Cited by:[§1](https://arxiv.org/html/2608.20447#S1.p1.1)\.
- \[13\]M\. A\. Hall and G\. Holmes\(2003\)Benchmarking attribute selection techniques for discrete class data mining\.IEEE Trans\. on Knowl\. and Data Eng\.15\(6\),pp\. 1437–1447\.External Links:ISSN 1041\-4347,[Link](http://dx.doi.org/10.1109/TKDE.2003.1245283),[Document](https://dx.doi.org/10.1109/TKDE.2003.1245283)Cited by:[§2\.1](https://arxiv.org/html/2608.20447#S2.SS1.p26.2)\.
- \[14\]J\. Han, M\. Kamber, and J\. Pei\(2011\)Data mining: concepts and techniques\.3rd edition,Morgan Kaufmann Publishers Inc\.,San Francisco, CA, USA\.External Links:ISBN 0123814790, 9780123814791Cited by:[§1](https://arxiv.org/html/2608.20447#S1.p6.1)\.
- \[15\]E\. Hancer, B\. Xue, M\. Zhang, D\. Karaboga, and B\. Akay\(2015\)A multi\-objective artificial bee colony approach to feature selection using fuzzy mutual information\.InEvolutionary Computation \(CEC\), 2015 IEEE Congress on,pp\. 2420–2427\.Cited by:[§2\.3](https://arxiv.org/html/2608.20447#S2.SS3.p3.1)\.
- \[16\]T\. Homma and A\. Saltelli\(1996\)Importance measures in global sensitivity analysis of nonlinear models\.Reliability Engineering & System Safety52\(1\),pp\. 1–17\.Cited by:[§2\.2](https://arxiv.org/html/2608.20447#S2.SS2.p1.1)\.
- \[17\]N\. Hoque, D\. Bhattacharyya, and J\. K\. Kalita\(2014\)MIFS\-nd: a mutual information\-based feature selection method\.Expert Systems with Applications41\(14\),pp\. 6371–6385\.Cited by:[§2\.3](https://arxiv.org/html/2608.20447#S2.SS3.p3.1)\.
- \[18\]A\. Idris and A\. Khan\(2012\)Customer churn prediction for telecommunication: employing various various features selection techniques and tree based ensemble classifiers\.InMultitopic Conference \(INMIC\), 2012 15th International,pp\. 23–27\.Cited by:[§2\.3](https://arxiv.org/html/2608.20447#S2.SS3.p3.1)\.
- \[19\]R\. Ihaka and R\. Gentleman\(1996\)R: a language for data analysis and graphics\.Journal of computational and graphical statistics5\(3\),pp\. 299–314\.Cited by:[§4](https://arxiv.org/html/2608.20447#S4.p6.1)\.
- \[20\]R\. H\. Kewley, M\. J\. Embrechts, and C\. Breneman\(2000\)Data strip mining for the virtual design of pharmaceuticals with neural networks\.Neural Networks, IEEE Transactions on11\(3\),pp\. 668–679\.Cited by:[§2\.2](https://arxiv.org/html/2608.20447#S2.SS2.p1.1)\.
- \[21\]Y\. Kim and W\. N\. Street\(2004\)An intelligent system for customer targeting: a data mining approach\.Decision Support Systems37\(2\),pp\. 215–228\.Cited by:[§2\.3](https://arxiv.org/html/2608.20447#S2.SS3.p1.1)\.
- \[22\]I\. Kondapaneni, P\. Kordík, and P\. Slavík\(2007\)Visualization techniques utilizing the sensitivity analysis of models\.InProceedings of the 39th conference on Winter simulation: 40 years\! The best is yet to come,pp\. 730–737\.Cited by:[§2\.2](https://arxiv.org/html/2608.20447#S2.SS2.p1.1)\.
- \[23\]H\. Liu and L\. Yu\(2005\)Toward integrating feature selection algorithms for classification and clustering\.Knowledge and Data Engineering, IEEE Transactions on17\(4\),pp\. 491–502\.Cited by:[§2\.3](https://arxiv.org/html/2608.20447#S2.SS3.p2.1)\.
- \[24\]Q\. Liu, Z\. Zhao, Y\. Li, and Y\. Li\(2012\)Feature selection based on sensitivity analysis of fuzzy isodata\.Neurocomputing85,pp\. 29–37\.Cited by:[§2\.2](https://arxiv.org/html/2608.20447#S2.SS2.p1.1)\.
- \[25\]S\. Moro, P\. Cortez, and P\. RitaA framework for increasing the value of predictive data\-driven models by enriching problem domain characterization with novel features\.Neural Computing and Applications,pp\. 1–9\.Cited by:[§1](https://arxiv.org/html/2608.20447#S1.p6.1),[§2\.2](https://arxiv.org/html/2608.20447#S2.SS2.p2.1)\.
- \[26\]S\. Moro, P\. Cortez, and P\. Rita\(2014\)A data\-driven approach to predict the success of bank telemarketing\.Decision Support Systems62,pp\. 22–31\.Cited by:[§1](https://arxiv.org/html/2608.20447#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.20447#S2.SS2.p2.1),[§2\.3](https://arxiv.org/html/2608.20447#S2.SS3.p2.1),[§3\.1](https://arxiv.org/html/2608.20447#S3.SS1.p1.1),[§3\.2](https://arxiv.org/html/2608.20447#S3.SS2.p2.1),[§4](https://arxiv.org/html/2608.20447#S4.p1.1),[§4](https://arxiv.org/html/2608.20447#S4.p7.1)\.
- \[27\]S\. Moro, P\. Cortez, and P\. Rita\(2015\)Using customer lifetime value and neural networks to improve the prediction of bank deposit subscription in telemarketing campaigns\.Neural Computing and Applications26\(1\),pp\. 131–139\.Cited by:[§2\.2](https://arxiv.org/html/2608.20447#S2.SS2.p2.1)\.
- \[28\]S\. Moro, P\. Rita, and B\. Vala\(2016\)Predicting social media performance metrics and evaluation of the impact on brand building: a data mining approach\.Journal of Business Research69\(9\),pp\. 3341–3351\.Cited by:[§2\.2](https://arxiv.org/html/2608.20447#S2.SS2.p2.1)\.
- \[29\]E\. W\. Ngai, L\. Xiu, and D\. C\. Chau\(2009\)Application of data mining techniques in customer relationship management: a literature review and classification\.Expert systems with applications36\(2\),pp\. 2592–2602\.Cited by:[§2\.3](https://arxiv.org/html/2608.20447#S2.SS3.p1.1)\.
- \[30\]F\. T\. Nobibon, R\. Leus, and F\. C\. Spieksma\(2011\)Optimization models for targeted offers in direct marketing: exact and heuristic algorithms\.European Journal of Operational Research210\(3\),pp\. 670–683\.Cited by:[§1](https://arxiv.org/html/2608.20447#S1.p2.1),[§2\.3](https://arxiv.org/html/2608.20447#S2.SS3.p1.1)\.
- \[31\]L\. Paninski\(2003\)Estimation of entropy and mutual information\.Neural computation15\(6\),pp\. 1191–1253\.Cited by:[§1](https://arxiv.org/html/2608.20447#S1.p4.1)\.
- \[32\]C\. A\. R\. Pinheiro and F\. McNeill\(2014\)Heuristics in analytics: a practical perspective of what influences our analytical world\.John Wiley & Sons\.Cited by:[§2\.3](https://arxiv.org/html/2608.20447#S2.SS3.p1.1)\.
- \[33\]P\. Refaeilzadeh, L\. Tang, and H\. Liu\(2009\)Cross\-validation\.InEncyclopedia of database systems,pp\. 532–538\.Cited by:[§3\.2](https://arxiv.org/html/2608.20447#S3.SS2.p2.1)\.
- \[34\]K\. A\. Richards and E\. Jones\(2008\)Customer relationship management: finding value drivers\.Industrial marketing management37\(2\),pp\. 120–130\.Cited by:[§2\.3](https://arxiv.org/html/2608.20447#S2.SS3.p1.1)\.
- \[35\]R\. Romero\-Meza, C\. Bonilla, H\. Benedetti, and A\. Serletis\(2015\)Nonlinearities and financial contagion in latin american stock markets\.Economic Modelling51,pp\. 653–656\.Cited by:[§2\.3](https://arxiv.org/html/2608.20447#S2.SS3.p2.1)\.
- \[36\]C\. Shannon\(1948\)A mathematical theory of communication\.Bell System Technical Journal27,pp\. 379–423, 623–656\.Cited by:[§2\.1](https://arxiv.org/html/2608.20447#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2608.20447#S2.SS1.p25.1)\.
- \[37\]D\. Tan, S\. Liew, T\. Tan, and W\. Yeoh\(2012\)A feature selection model for binary classification of imbalanced data based on preference for target instances\.InData Mining and Optimization \(DMO\), 2012 4th Conference on,pp\. 35–42\.Cited by:[§1](https://arxiv.org/html/2608.20447#S1.p6.1)\.
- \[38\]D\. Tan, W\. Yeoh, Y\. L\. Boo, and S\. Liew\(2013\)The impact of feature selection: a data\-mining application in direct marketing\.Intelligent Systems in Accounting, Finance and Management20\(1\),pp\. 23–38\.Cited by:[§1](https://arxiv.org/html/2608.20447#S1.p6.1),[§1](https://arxiv.org/html/2608.20447#S1.p7.1)\.
- \[39\]J\. A\. B\. Tinoco, A\. G\. Correia, and P\. Cortez\(2012\)Application of a sensitivity analysis procedure to interpret uniaxial compressive strength prediction of jet grouting laboratory formulations performed by svm model\.Cited by:[§2\.2](https://arxiv.org/html/2608.20447#S2.SS2.p2.1)\.
- \[40\]E\. Turban, R\. Sharda, and D\. Delen\(2011\)Decision support and business intelligence systems, 9th edition\.Pearson\.Cited by:[§1](https://arxiv.org/html/2608.20447#S1.p1.1)\.
- \[41\]C\. Vajiramedhin and A\. Suebsing\(2014\)Feature selection with data balancing for prediction of bank telemarketing\.Applied Mathematical Sciences8\(114\),pp\. 5667–5672\.Cited by:[§3\.1](https://arxiv.org/html/2608.20447#S3.SS1.p1.1)\.
- \[42\]T\. Verbraken, W\. Verbeke, and B\. Baesens\(2014\)Profit optimizing customer churn prediction with bayesian network classifiers\.Intelligent Data Analysis18\(1\),pp\. 3–24\.Cited by:[§2\.3](https://arxiv.org/html/2608.20447#S2.SS3.p3.1),[§3\.1](https://arxiv.org/html/2608.20447#S3.SS1.p1.1)\.
- \[43\]I\.H\. Witten, E\. Frank, and M\.A\. Hall\(2011\)Data mining: practical machine learning tools and techniques\.3rd edition,Morgan Kaufmann\.Cited by:[§1](https://arxiv.org/html/2608.20447#S1.p1.1),[§2\.3](https://arxiv.org/html/2608.20447#S2.SS3.p2.1)\.
- \[44\]X\. Wu, X\. Zhu, G\. Wu, and W\. Ding\(2014\)Data mining with big data\.Knowledge and Data Engineering, IEEE Transactions on26\(1\),pp\. 97–107\.Cited by:[§1](https://arxiv.org/html/2608.20447#S1.p2.1)\.

## The authors

### Nestor R\. Barraza

received his Msc\. and Ph\.D\. in Electronic Engineering from the University of Buenos Aires in 1993 and 1998\. He is currently a full Professor in the National University of Tres de Febrero, Argentina\. Hi is also with the School of Engineering of the University of Buenos Aires\. He has authored and co\-authored many scientific and academic papers and participated in several local and international conferences\. His research interests include Information Theory and Coding, Communications, Business Intelligence and Software Reliability\.

### Sérgio Moro

is an Assistant Professor at Instituto Universitário de Lisboa \(ISCTE\-IUL\), and member of ISTAR\-IUL and ALGORITMI Research Center\. He holds a PhD in Information Sciences and Technologies and an MSc in Management Information Systems, both from ISCTE\-IUL, and a 5 year BSc in Computer Engineering from Instituto Superior Técnico \(University of Lisbon\)\. His research appears in journals such as Decision Support Systems, Expert Systems with Applications, and Journal of Business Research\. He has worked for 15 years \(2001\-2016\) at Montepio Bank, as a Software Engineer, Project Manager and Business Intelligence & Analytics Manager\.

### Marcelo Ferreyra

is the founder of powerhouse, a Data Mining software tool\. He is currently CEO at dataxplore and independent consultant of Business Intelligence and Data Mining\.

### Adolfo de la Peña

received his Msc\. in Electronic Engineering from the University of Buenos Aires in 1990 and his Msc\. in Business Administration from the University of Salvador in 2013\. He is currently Engineering Manager at the Computer Department at Boldt Gaming S\.A\. His research interest includes Business Intelligence and Data Mining\.