利用局部内在维度进行无监督特征选择——扩展版

# Unsupervised Feature Selection Using Local Intrinsic Dimensionality – Extended Version
Source: [https://arxiv.org/html/2610.40736](https://arxiv.org/html/2610.40736)
Michael E\. Houleorganization=New Jersey Institute of Technology, country=United StatesKen\-ichi Kawarabayashiorganization=National Institute of Informatics, country=Japanorganization=University of Tokyo, country=JapanMuhammad RajabinasabArthur Zimekorganization=University of Southern Denmark, country=Denmark

###### Abstract

High dimensionality degrades both the efficiency and the effectiveness of learning algorithms, and in many applications no class labels are available to guide dimensionality reduction\. This paper proposes LIDFS, an unsupervised feature selection method based on Local Intrinsic Dimensionality \(LID\)\. The method builds on the formal equivalence between LID and the indiscriminability of distance measures\. The data matrix is transposed so that each feature is treated as a point, and the LID of each feature is estimated by maximum likelihood from the distances to itskknearest neighboring features\. Features with lower LID exhibit greater local discriminability and receive higher importance scores\. A refinement step then retains one representative per feature neighborhood and demotes locally redundant features\. The result is a complete ranking from which any number of features can be selected without rerunning the procedure\. LIDFS does not use any information about labels or the number of classes, and runs in𝒪⁡\(d2​n\)\\mathcal\{O\}\(d^\{2\}n\)time fornninstances andddfeatures\. LIDFS is compared with classic and state\-of\-the\-art methods and a random baseline on 23 high\-dimensional datasets, the classic benchmark collection\. The evaluation uses classification accuracy and AUC with five downstream classifiers, as well as clustering accuracy and normalized mutual information, over both full and aggressive selection ranges\. LIDFS ranks first in the supervised evaluations and first or tied second in the clustering\-based evaluations\. It belongs to the top statistical group together with methods that require the number of classes, and it runs faster than its closest competitor\. Several recent methods, by contrast, fail to outperform the random baseline\. A sensitivity analysis shows that small neighborhood sizes yield the best results\.

###### keywords

Unsupervised Feature Selection ,Local Intrinsic Dimensionality ,Dimensionality Reduction ,Big Data

††corresponding:Corresponding author: rajabinasab@imada\.sdu\.dk## 1Introduction

Advances in computer technology are opening the way for the handling of increasingly complex data\. The scale of data can be measured with respect to volume \(the number of data instances\) and dimensionality \(the number of features that describe these instances\)\. When dealing with colossal data volume on high\-capacity computing platforms, volume reduction techniques such as sampling, together with parallelization, can allow the exploration and analysis of data with millions or even billions of data entries\.

Feature reduction, the dimensional analogue of volume reduction, has several motivations\[[1](https://arxiv.org/html/2610.40736#bib.bib1)\]: reducing the costs of data collection, storage, and processing; improving the model generalizability and interpretability; and improving the discriminative power of feature\-based distance measures\. In general, high dimensionality is associated with a degradation in the efficiency and effectiveness of fundamental data mining and machine learning tasks such as clustering and anomaly detection\[[2](https://arxiv.org/html/2610.40736#bib.bib2)\], an effect often referred to as the ‘curse of dimensionality’\. As the dimensionality rises, distance values between points tend to concentrate around their mean values due to the effects of noise and other sources of error, which drastically diminishes the discriminative power of the distance measure\.

Feature reduction methods fall into two main categories: selection and extraction\. Feature selection, as the name suggests, retains a subset of the original features, chosen with the aim of improving the performance and efficiency of data mining and machine learning tasks\[[3](https://arxiv.org/html/2610.40736#bib.bib3)\]\. In feature extraction, the attributes are transformed to a smaller set within an artificial feature space, with each new feature value depending on the values of many \(or even all\) of the original features\. Many feature extraction algorithms are based on Principal Component Analysis \(PCA\)\[[4](https://arxiv.org/html/2610.40736#bib.bib4)\]or Linear Discriminant Analysis\[[5](https://arxiv.org/html/2610.40736#bib.bib5)\]\.

For applications in which extraction methods are appropriate, extracted features are very likely to outperform any equal number of selected features, due to the greater flexibility in determining a data transformation that maximizes the quality criteria used to guide the feature generation process\. However, in some contexts where feature semantics are of major importance and where artificial features are meaningless, feature extraction cannot be used\. Such applications are common in bioinformatics\[[6](https://arxiv.org/html/2610.40736#bib.bib6)\]\(gene annotation, micro\-array analysis\) and chemistry\[[6](https://arxiv.org/html/2610.40736#bib.bib6),[7](https://arxiv.org/html/2610.40736#bib.bib7)\]\(mass spectra analysis, interpretability of molecular signatures\), among others\. In such contexts, however, feature selection can still be applied to reduce the dimensionality of the data\.

Even when feature extraction methods are applicable, they are generally not suitable for very large datasets, as the transformation of the full dataset to a new full\-dimensional basis can incur a prohibitively high computational cost\. This computational complexity, together with a lack of robustness in the presence of noise\[[8](https://arxiv.org/html/2610.40736#bib.bib8)\], makes feature extraction less useful when the instances outnumber the features\. Furthermore, feature extraction has limited use when the goal is to remove irrelevant features, since all of the original features must be retained in order to compute values within the transformed feature space\. Feature extraction also reduces the explainability and transparency of the final downstream task as it becomes difficult to interpret the results which rely on a transformed space\.

In this paper, we consider the case of unsupervised feature selection, where class label information is unavailable\. Unsupervised feature selection is a more difficult problem than supervised feature selection, due to the difficulty in evaluating the quality of features\. In the absence of ground truth information, there are two main types of quality criteria that are commonly used to guide the selection process: redundancy elimination, where an attribute is discarded whenever it can be entirely or partially inferred from other attributes, and similarity conservation, in which a feature is selected according to the degree to which it helps to conserve the similarity between data points\. Of the two, similarity conservation has received relatively less attention in the research literature\.

Depending on their evaluation strategy, feature selection frameworks can be categorized as either filters or wrappers \(or embedded\)\. Wrapper methods execute a predictive learning task over the data for many choices of feature subsets, and retain candidate features for which the predictive model achieves high performance\[[1](https://arxiv.org/html/2610.40736#bib.bib1),[9](https://arxiv.org/html/2610.40736#bib.bib9)\]\. In practical contexts involving data of even moderate volume or dimension, the cost of wrapper methods quickly becomes prohibitively high\. A more affordable alternative is that of filter methods, which evaluate feature quality without resorting to an external learning machine\. Although filters generally cannot match the quality of wrappers on small\-scale datasets, on datasets of larger scale filters generally require far less computation time, and are less susceptible to overfitting\. Many unsupervised feature selection algorithms \(such as LapAOFS / LapDOFS\[[10](https://arxiv.org/html/2610.40736#bib.bib10)\]\) require that a target number of features be supplied as an input parameter, while others \(such as GLFS\[[11](https://arxiv.org/html/2610.40736#bib.bib11)\]\) automatically determine the number of selected features\. With algorithms of the former type, the user is free to increase the number of features if needed, although the entire feature selection task usually must be reiterated\. On the other hand, with most algorithms of the latter type, the user has no control over the number of features selected\. When dealing with large datasets, a third and more convenient alternative is to produce a ranked list of all features, so as to allow any downstream processes to select the number of features that are most appropriate to the task at hand, given the amount of computational resources available\. The most widely known unsupervised feature ranking framework is that of spectral feature selection\[[12](https://arxiv.org/html/2610.40736#bib.bib12)\], which includes Laplacian Score for Feature Selection \(LS\)\[[13](https://arxiv.org/html/2610.40736#bib.bib13)\]\. LS attempts to produce a reduced feature set that preserves the original neighborhood information to the greatest possible extent\. More recent work on filters for unsupervised feature ranking includes Multi\-Cluster Feature Selection \(MCFS\)\[[14](https://arxiv.org/html/2610.40736#bib.bib14)\], which attempts to preserve the cluster structure of the data\.

Almost all existing filter methods for feature selection are suitable only for datasets of relatively small volume and dimensionality, such as those found in the context of bioinformatics\. In this work, we target the problem of unsupervised feature ranking at higher scales, where the data volume and data dimension together preclude the use of wrapper methods and computationally expensive filters\. Such feature selection could be of particular benefit for database and data mining operations such as similarity search, neighborhood\-based classification, and clustering\.

As a criterion for guiding the selection process, we propose that features be assessed according to their ability to discriminate between the distances encountered in the neighborhoods of data points, in an attempt to alleviate the effects of the curse of dimensionality\. Accordingly, we develop a feature ranking method that assesses the discriminative power of individual features according to a model of the local intrinsic dimensionality \(LID\) of continuous distance distributions\[[15](https://arxiv.org/html/2610.40736#bib.bib15),[16](https://arxiv.org/html/2610.40736#bib.bib16),[17](https://arxiv.org/html/2610.40736#bib.bib17)\]\. While most feature selection methods employ indirect strategies in order to \(implicitly\) improve the discriminability of distance measures, the use of LID as guiding criterion allows for a more explicit approach, due to a formal equivalence\[[16](https://arxiv.org/html/2610.40736#bib.bib16)\]between the discriminability of distance measures and low intrinsic dimensionality\. Taking into account the performance issues due to larger data scales \(in terms of both volume and dimensionality\), we develop a forward filter algorithm generally applicable to continuous\-valued numerical datasets\.

The specific contributions of this paper include:

- •a greedy unsupervised feature selection algorithm based on LID111Python implementation and benchmark code are available at[https://github\.com/mrajabinasab/LIDFS/](https://github.com/mrajabinasab/LIDFS/)\.;
- •a theoretical analysis and discussion of the proposed method;
- •an experimental framework setting random feature selection as a baseline for feature selection, and showing the advantage of using our method with high\-dimensional feature selection benchmark datasets\.

The paper is organized as follows\. In the next section we present a survey of existing methods for feature selection\. In Section[3](https://arxiv.org/html/2610.40736#S3), we introduce the LID model upon which our feature ranking method is based, as well as its relationship to the discriminability of distance measures\. In Section[4](https://arxiv.org/html/2610.40736#S4), we present the details of the proposed unsupervised feature selection method\. The experimental framework is presented in Section[5](https://arxiv.org/html/2610.40736#S5), in which our method is compared against state\-of\-the\-art unsupervised feature selection algorithms\. The experimental results are discussed in Section[6](https://arxiv.org/html/2610.40736#S6)\. Finally, in Section[7](https://arxiv.org/html/2610.40736#S7)we conclude by summarizing the advantages and the limitations of our feature selection strategy, and propose some possible extensions of our algorithm as future work\.

## 2Feature selection

Feature selection consists of choosing a subset of the original features so as to best satisfy a quality criterion\. The ultimate goal is to improve model generalizability and interpretability, as well as to lower the computational costs of learning algorithms that use the data\.

Formally, let us assume that we are given a set ofnnpoints in a space of dimensionddrepresented by the matrixX∈Mn,d​\(ℝ\)X\\in M\_\{n,d\}\(\\mathbb\{R\}\),

X=\(x1,x2,…,xn\)⊤=\(f1,f2,…,fd\),X=\(x\_\{1\},x\_\{2\},\\ldots,x\_\{n\}\)^\{\\top\}=\(f\_\{1\},f\_\{2\},\\ldots,f\_\{d\}\),wherexi∈ℝdx\_\{i\}\\in\\mathbb\{R\}^\{d\}are points andfi∈ℝnf\_\{i\}\\in\\mathbb\{R\}^\{n\}are features\. Feature selection consists of findingΩ′=\{f1′,…,fd′′\}\\Omega^\{\\prime\}=\\\{f^\{\\prime\}\_\{1\},\\ldots,f^\{\\prime\}\_\{d^\{\\prime\}\}\\\}, a subset ofΩ=\{f1,…,fd\}\\Omega=\\\{f\_\{1\},\\ldots,f\_\{d\}\\\}withd′<dd^\{\\prime\}<d, that best satisfies an optimality condition\.

Feature selection algorithms can be differentiated in several ways\. They can be

- •wrappers or filters, according to whether or not they employ a learning task as part of their feature evaluation strategy \(with embedded methods combining the characteristics of both\);
- •forward or backward depending on their search direction, whether they build a feature set incrementally starting from an empty set, or cull features incrementally starting from the full set;
- •univariate or multivariate, according to whether they assess features individually or in groups\.

Wrapper methods apply a predictive learning machine to the data using the candidate feature subsets; the objective function is defined by the performance of this predictive model on a task for which the feature selection is targeted\[[9](https://arxiv.org/html/2610.40736#bib.bib9)\]\. Wrappers can achieve high performance with the learning machine used in the selection process\. However, they are model\-specific\[[1](https://arxiv.org/html/2610.40736#bib.bib1)\], as features selected using a given learning model are often inadequate for others\. Wrapper methods are also computationally expensive\[[1](https://arxiv.org/html/2610.40736#bib.bib1)\]: in a wrapper selection process, assessing the quality of each candidate subset requires one run of the learning machine, and one measurement of the quality of its output\. Moreover, wrappers present a risk of overfitting not only to the predictive model, but also to the data\. In contrast, filter methods use properties of the data to compare the candidate subsets: they evaluate candidate feature subsets in terms of some quality criterion, often information\-theoretic \(MRMR\[[18](https://arxiv.org/html/2610.40736#bib.bib18)\], Information Gain\[[19](https://arxiv.org/html/2610.40736#bib.bib19)\]\), correlation\-based \(Correlation\-based Feature Selection\[[20](https://arxiv.org/html/2610.40736#bib.bib20)\]\), statistical \(t\-test and chi\-squared\[[21](https://arxiv.org/html/2610.40736#bib.bib21)\]\), or based on interclass distances \(Fisher Score\[[22](https://arxiv.org/html/2610.40736#bib.bib22)\], supervised Spectral Feature Selection\[[12](https://arxiv.org/html/2610.40736#bib.bib12)\]\)\. Although they cannot match the performance of wrappers on the learning tasks used to guide the feature selection process, filters have the advantage of not being specific to any particular learning model\. Furthermore, filters generally require far less computation time, and are less susceptible to overfitting\. In the context of complex data, the advantages of filters are crucial\.

Feature selection algorithms can also be categorized according to the order in which candidate feature subsets are considered\. With backward elimination, the first feature subset candidate assessed is the full datasetXX; new candidate subsets are constructed by incrementally removing features that underperform with respect to some criterion\. The opposite approach, forward selection, consists of starting with an empty set, and building up candidate feature subsets through the incremental introduction of individual features\. When the number of features is very high, examining candidate subsets of smaller cardinality is more affordable\. In the context of high\-dimensional data, this argument is largely in favor of forward selection\.

Feature selection methods also differ in the way they group features for assessment: univariate methods assess features individually and independently, while multivariate methods consider sets of features during the selection process\. Since combinations of features are not considered, the univariate approach cannot account for redundancy of features\. However, univariate methods have the general advantage of being faster: since they consider attributes separately, their complexity is usually linear in the number of features\. Although the multivariate approach is usually more computationally expensive, in the context of high\-dimensional data, multivariate methods that strictly limit the numbers of candidates assessed can sometimes be more affordable than their univariate competitors\.

In some cases, a vectorL=\(l1,l2,…,ln\)L=\(l\_\{1\},l\_\{2\},\.\.\.,l\_\{n\}\)of class labels associated withXXmay also be available\. Feature selection methods that make use of such label information as a ground truth are said to be ‘supervised’; if label information is not used, the method is said to be ‘unsupervised’\. In general, wrapper methods tend to be supervised \(due to the need to assess the result on a specific learning task\)\.

In the context of big data, where ground truth label information is rarely available, the complexity argument favors unsupervised feature selection filters which operate in a forward direction\. Among all existing candidates that fit these criteria, scalability concerns prevent the use of computationally expensive algorithms such as LapDOFS and LapAOFS\[[10](https://arxiv.org/html/2610.40736#bib.bib10)\], both of which require time more than quadratic in the number of points and number of features\. Among all algorithms with reasonable scalability characteristics, the unsupervised general\-purpose forward\-filter Laplacian Score \(LS\) feature selection method\[[13](https://arxiv.org/html/2610.40736#bib.bib13)\]is perhaps the most popular\. It belongs to the general framework of spectral feature selection\[[12](https://arxiv.org/html/2610.40736#bib.bib12)\], and has a computational complexity of𝒪⁡\(d​n2\)\\mathcal\{O\}\(dn^\{2\}\)\. The spectral framework offers three different approaches to scoring features, with the objective always being to preserve the graph structure of the dataset described by a spectral matrixSS\. Of the three scoring functions available, the second is used by LS:

𝑆𝑐𝑜𝑟𝑒LS​\(ft\)\\displaystyle\\mathit\{Score\}\_\{\\mathrm\{LS\}\}\(f\_\{t\}\)=\\displaystyle=∑xi,xj∈X\(xi,t−xj,t\)2𝖵𝖺𝗋⁡\(ft\)​si,j,\\displaystyle\\sum\_\{x\_\{i\},x\_\{j\}\\in X\}\\frac\{\(x\_\{i,t\}\-x\_\{j,t\}\)^\{2\}\}\{\\mathsf\{Var\}\(f\_\{t\}\)\}s\_\{i,j\},\(1\)where​si,j\\displaystyle\\textrm\{ where \}\\\>\\\>\\\>s\_\{i,j\}=\\displaystyle=δi,j​e−di,j2θ;\\displaystyle\\delta\_\{i,j\}e^\{\-\\frac\{d\_\{i,j\}^\{2\}\}\{\\theta\}\};\(2\)𝖵𝖺𝗋⁡\(ft\)\\mathsf\{Var\}\(f\_\{t\}\)is the variance of the featureftf\_\{t\};di,jd\_\{i,j\}denotes the Euclidean distance betweenxix\_\{i\}andxjx\_\{j\};δi,j\\delta\_\{i,j\}is11whenxix\_\{i\}orxjx\_\{j\}is among thekk\-nearest neighbors of the other, and 0 otherwise; andθ\\thetarepresents the variance within a set of distance values, but is not explicitly defined in the original work\. Features are ranked in ascending order of𝑆𝑐𝑜𝑟𝑒LS\\mathit\{Score\}\_\{\\mathrm\{LS\}\}\.

Feature selection can also be guided according to how well the new features preserve the cluster structure of the data\. Given a similarity graph represented as a matrixWWwith\(W\)i,j=δi,j\(W\)\_\{i,j\}=\\delta\_\{i,j\}, and a target number of clusterscc, Multi\-Cluster Feature Selection \(MCFS\)\[[14](https://arxiv.org/html/2610.40736#bib.bib14)\]computes thecceigenvectors\(y1,…,yc\)\(y\_\{1\},\\ldots,y\_\{c\}\)with the smallest eigenvalues of the generalized eigenproblemL​y=λ​D​yLy=\\lambda Dy, whereDDis the diagonal matrix\(D\)i,i=∑j\(W\)i,j\(D\)\_\{i,i\}=\\sum\_\{j\}\(W\)\_\{i,j\}, andL=D−WL=D\-W\. Least Angle Regression \(LAR\) is then used to solve theccregression problems

minai⁡‖yi−X​ai‖2,subject to​‖ai‖1≤γ\.\\min\_\{a\_\{i\}\}\\\|y\_\{i\}\-Xa\_\{i\}\\\|^\{2\},\\\>\\\>\\\>\\textrm\{ subject to \}\\\>\\\>\\\>\\\|a\_\{i\}\\\|\_\{1\}\\leq\\gamma\.\(3\)Finally, features are scored by

𝑆𝑐𝑜𝑟𝑒MCFS​\(ft\)=maxi⁡\|ai,t\|\.\\mathit\{Score\}\_\{\\mathrm\{MCFS\}\}\(f\_\{t\}\)=\\max\_\{i\}\|a\_\{i,t\}\|\.\(4\)Features are ranked in descending order of𝑆𝑐𝑜𝑟𝑒MCFS\\mathit\{Score\}\_\{\\mathrm\{MCFS\}\}\. MCFS has limited use in practice due to its high computational complexity:𝒪⁡\(d​n2\+c​d3\+c​n​d2\)\\mathcal\{O\}\(dn^\{2\}\+cd^\{3\}\+cnd^\{2\}\)\.

Structured Optimal Graph Feature Selection \(SOGFS\)\[[23](https://arxiv.org/html/2610.40736#bib.bib23)\]jointly learns a probabilistic\-neighbor similarity graph and a row\-sparse projection matrix, rather than fixing the graph from the raw data beforehand\. To keep the graph well\-structured, it constrains the Laplacian𝐋𝐒\\mathbf\{L\_\{S\}\}to have exactlyccconnected components, relaxed via Ky Fan’s theorem into a trace\-minimization term,

∑i=1cσi​\(𝐋𝐒\)=min𝐅∈ℝn×c,𝐅⊤​𝐅=𝐈⁡Tr⁡\(𝐅⊤​𝐋𝐒​𝐅\),\\sum\_\{i=1\}^\{c\}\\sigma\_\{i\}\(\\mathbf\{L\_\{S\}\}\)=\\min\_\{\\mathbf\{F\}\\in\\mathbb\{R\}^\{n\\times c\},\\,\\mathbf\{F\}^\{\\top\}\\mathbf\{F\}=\\mathbf\{I\}\}\\mathrm\{Tr\}\(\\mathbf\{F\}^\{\\top\}\\mathbf\{L\_\{S\}\}\\mathbf\{F\}\),\(5\)solved by alternating updates of the projection,𝐅\\mathbf\{F\}, and the graph, at complexity𝒪⁡\(d3\+n2​m\)\\mathcal\{O\}\(d^\{3\}\+n^\{2\}m\), wheremmis the projection dimensionality\.

Local structure Learning and exponentially weighted Sparse Regression Feature Selection \(LLSRFS\)\[[24](https://arxiv.org/html/2610.40736#bib.bib24)\]counters the tendency of sparse mapping matrices to overweight already\-high\-scoring features by reweighting each feature’s contribution to the pairwise distance bywlqw\_\{l\}^\{q\},

‖𝐱i−𝐱j‖e​w2=∑l=1dwlq​\(xi​l−xj​l\)2,\\\|\\mathbf\{x\}\_\{i\}\-\\mathbf\{x\}\_\{j\}\\\|\_\{ew\}^\{2\}=\\sum\_\{l=1\}^\{d\}w\_\{l\}^\{q\}\(x\_\{il\}\-x\_\{jl\}\)^\{2\},\(6\)where a smaller exponentq∈\(0,1\]q\\in\(0,1\]flattens the weight distribution across more features\. The authors show this is equivalent toℓ2,p2\\ell\_\{2,p\}^\{2\}\-norm sparse regression underq=1/p−1/2q=1/p\-1/2, generalizing earlier rescaled\-least\-squares selectors, at complexity𝒪⁡\(d3\+n2​d\+n​d2\)\\mathcal\{O\}\(d^\{3\}\+n^\{2\}d\+nd^\{2\}\)\.

Subspace Clustering unsupervised Feature Selection \(SCFS\)\[[25](https://arxiv.org/html/2610.40736#bib.bib25)\]implicitly learns sample similarity through a self\-expressive representation rather than an explicit graph,

min𝐆⁡‖𝐗−𝐆𝐆⊤​𝐗‖F2,s\.t\.​𝐆≥0,𝐆𝐆⊤​𝟏=𝟏,\\min\_\{\\mathbf\{G\}\}\\\|\\mathbf\{X\}\-\\mathbf\{G\}\\mathbf\{G\}^\{\\top\}\\mathbf\{X\}\\\|\_\{F\}^\{2\},\\quad\\text\{s\.t\. \}\\mathbf\{G\}\\geq 0,\\ \\mathbf\{G\}\\mathbf\{G\}^\{\\top\}\\mathbf\{1\}=\\mathbf\{1\},\(7\)where the nonnegative𝐆\\mathbf\{G\}doubles as a soft clustering assignment, so𝐆𝐆⊤\\mathbf\{G\}\\mathbf\{G\}^\{\\top\}encodes pairwise similarity implicitly\. This is linked to the features by a sparse regressionmin𝐖⁡‖𝐗𝐖−𝐆‖F2\+β​‖𝐖‖2,1\\min\_\{\\mathbf\{W\}\}\\\|\\mathbf\{X\}\\mathbf\{W\}\-\\mathbf\{G\}\\\|\_\{F\}^\{2\}\+\\beta\\\|\\mathbf\{W\}\\\|\_\{2,1\}, optimized jointly with the first objective, at complexitymax⁡\{𝒪⁡\(d3\),𝒪⁡\(n2​d\)\}\\max\\\{\\mathcal\{O\}\(d^\{3\}\),\\mathcal\{O\}\(n^\{2\}d\)\\\}whenc≪dc\\ll d\.

Variance–Covariance Subspace Distance Feature Selection \(VCSDFS\)\[[26](https://arxiv.org/html/2610.40736#bib.bib26)\]selects features by minimizing a proper metric distance between the full feature matrix and a candidate sub\-matrix,

dist⁡\(𝐗,𝐗I\)=‖𝐗𝐗⊤−𝐗I​𝐗I⊤‖F,\\mathrm\{dist\}\(\\mathbf\{X\},\\mathbf\{X\}\_\{I\}\)=\\\|\\mathbf\{XX\}^\{\\top\}\-\\mathbf\{X\}\_\{I\}\\mathbf\{X\}\_\{I\}^\{\\top\}\\\|\_\{F\},\(8\)which the authors show equals the Frobenius norm of the covariance matrix of the discarded features, so minimizing it discards the least\-variance features\. This avoids the degeneracy of principal\-angle\-based distances, which are provably zero between any matrix and its own sub\-matrices\. The feature weights are learned via a matrix\-factorization objective with an inner\-product regularizer, at complexity𝒪⁡\(t​d2\)\\mathcal\{O\}\(td^\{2\}\), wherettis the number of iterations\.

Feature selection with Minkowski weightedkk\-means\+\+ \(FS\-MWK\+\+\)\[[27](https://arxiv.org/html/2610.40736#bib.bib27)\]selects centroids probabilistically using feature relevance estimates derived from the data, and then aggregates feature weights across a range of Minkowski exponents to identify informative features\. A scalable variant of the method, namely SFS\-MWK\+\+, is based on subsampling\. Robust, Adaptive and Flexible Graph \(RAFG\)\[[28](https://arxiv.org/html/2610.40736#bib.bib28)\]is a graph\-learning framework for unsupervised feature selection\. TheL2,1L\_\{2,1\}\-norm is imposed on the flexible regression term to alleviate the adverse effects of both noisy features and outliers, and anL2,1L\_\{2,1\}\-norm regularization term is incorporated to ensure that the selected transformation matrix is sufficiently sparse\. Several other methods proposed in recent years focus on the unsupervised*multi\-view*feature selection problem\[[29](https://arxiv.org/html/2610.40736#bib.bib29),[30](https://arxiv.org/html/2610.40736#bib.bib30)\]\.

## 3Indiscriminability and Local ID

The success of learning algorithms on high\-dimensional data is often attributed to a substantial gap between the high representational dimension of the data and its comparatively low intrinsic dimensionality \(ID\)\[[16](https://arxiv.org/html/2610.40736#bib.bib16),[17](https://arxiv.org/html/2610.40736#bib.bib17)\]\. Many feature extraction and feature selection methods implicitly exploit this gap in order to improve discriminability\. However, intrinsic dimensionality may vary considerably across different regions or clusters of a dataset\. Such local variation in ID is seldom exploited explicitly in feature selection\.

### 3\.1Expansion Dimension and Local ID

Many models of ID have been proposed over the last few decades: for the theoretical analysis and characterization of datasets, the most popular of these have been topological models, such as the well\-known PCA\[[4](https://arxiv.org/html/2610.40736#bib.bib4)\]; fractal models\[[31](https://arxiv.org/html/2610.40736#bib.bib31)\]; covering models\[[32](https://arxiv.org/html/2610.40736#bib.bib32)\]; and expansion models\[[33](https://arxiv.org/html/2610.40736#bib.bib33),[34](https://arxiv.org/html/2610.40736#bib.bib34)\]\. Most of the measures of ID associated with these models are very expensive to compute, and are thus of limited use in practice\. These models can be used ‘globally’ across the entire dataset, or adapted ‘locally’ so as to characterize the local dimensionality in the vicinity of reference points\.

Global estimates of ID are typically higher than local estimates, since the number of degrees of freedom required to describe a dataset is \(as a rule\) higher than that required to describe its components\. This gap between global and local ID measurements indicates an opportunity for the latter in guiding feature selection to improve the performance of similarity search\. Improvements in the performance of search would have a direct impact on applications of machine learning, data mining, and databases\. Among the models of ID mentioned above, expansion models have seen use in the design and analysis of similarity search algorithms\[[34](https://arxiv.org/html/2610.40736#bib.bib34),[35](https://arxiv.org/html/2610.40736#bib.bib35),[36](https://arxiv.org/html/2610.40736#bib.bib36),[37](https://arxiv.org/html/2610.40736#bib.bib37)\], as well as outlier detection\[[38](https://arxiv.org/html/2610.40736#bib.bib38),[39](https://arxiv.org/html/2610.40736#bib.bib39)\]\.

The intrinsic dimensionalitymmin a Euclidean vector space can be determined using the ratios of two volume and two distance measurements\. Consider the concentric ballsB⁡\(x,r1\)B\(x,r\_\{1\}\)andB⁡\(x,r2\)B\(x,r\_\{2\}\)centered atxxwith radii0<r1<r20<r\_\{1\}<r\_\{2\}\. Letλ\\lambdabe a \(Lebesgue\) volume measure\. The ratio of volumes leads to a simple closed\-form expression for the dimensionalityddof Euclidean spaces:

λ⁡\(B⁡\(x,r2\)\)λ⁡\(B⁡\(x,r1\)\)=\(r2r1\)m⟹m=ln⁡λ⁡\(B⁡\(x,r2\)\)−ln⁡λ⁡\(B⁡\(x,r1\)\)ln⁡r2−ln⁡r1\.\\begin\{split\}&\\frac\{\\lambda\(B\(x,r\_\{2\}\)\)\}\{\\lambda\(B\(x,r\_\{1\}\)\)\}=\{\\Bigg\(\\frac\{r\_\{2\}\}\{r\_\{1\}\}\\Bigg\)\}^\{m\}\\\\ \\Longrightarrow\\,\\,\\,\\,&m=\\frac\{\\ln\\lambda\(B\(x,r\_\{2\}\)\)\-\\ln\\lambda\(B\(x,r\_\{1\}\)\)\}\{\\ln r\_\{2\}\-\\ln r\_\{1\}\}\\,\.\\end\{split\}\(9\)An oracle that accepts two concentric balls as a query, and returns the dimension of the space, is not immediately useful\. However, if volume is estimated by the number of points within distancerrfrom a reference pointxx, then the returned value can serve as a measure of intrinsic dimensionality, the local expansion dimension \(ED\)\[[33](https://arxiv.org/html/2610.40736#bib.bib33),[34](https://arxiv.org/html/2610.40736#bib.bib34)\]\.

The model of intrinsic dimensionality considered here is a variant of ED, in which the notion of volume is generalized to a probability measure, and distances are modeled as a random variable\[[16](https://arxiv.org/html/2610.40736#bib.bib16),[17](https://arxiv.org/html/2610.40736#bib.bib17)\]\. Letxxbe a reference point, andXXbe a random variable expressing a distance measurement fromXX\. LetPXP\_\{X\}be the cumulative distribution function associated withXX\. For anyrrsuch thatPX​\(r\)\>0P\_\{X\}\(r\)\>0andPXP\_\{X\}is continuously differentiable in an interval containingrr, the Local Intrinsic Dimensionality \(LID\) ofXXat distancerris defined, wherever the limit exists, as

LIDPX​\(r\)≜limϵ→0\+ln⁡\(PX​\(\(1\+ϵ\)​r\)/PX​\(r\)\)ln⁡\(1\+ϵ\)=r⋅PX′​\(r\)PX​\(r\)\\mathrm\{LID\}\_\{P\_\{X\}\}\(r\)\\triangleq\\\!\\\!\\\!\\lim\\limits\_\{\\epsilon\\to 0^\{\+\}\}\\\!\\\!\\frac\{\\mathrm\{ln\}\(P\_\{X\}\(\(1\+\\epsilon\)r\)/P\_\{X\}\(r\)\)\}\{\\mathrm\{ln\}\(1\+\\epsilon\)\}=\\frac\{r\\cdot P^\{\\prime\}\_\{X\}\(r\)\}\{P\_\{X\}\(r\)\}\(10\)

### 3\.2Indiscriminability

A distance measure is ‘discriminative’ when an expansion in distance results in a relatively small increase in the number of observations\. Discriminability of points improves the efficiency of data mining and machine learning tasks, minimizes error and ensures robustness\. Capturing a huge number of points by a small increase in distance measure greatly increases the computational cost of applications, and has a negative impact on effectiveness\. Moreover, the relative ranking of points with reference to an indiscriminative distance measure can easily be affected by noise\. When selecting features, it is strongly desirable that they be chosen so as to ensure better discriminability\.

Formally, letxxbe a reference point and letXXbe a random distance variable with respect to that reference point, as defined in Section[3\.1](https://arxiv.org/html/2610.40736#S3.SS1)\. For any distancerrfromxxsuch thatPX​\(r\)\>0P\_\{X\}\(r\)\>0, the indiscriminability ofXXatrris given by the following limit wherever it exists\[[16](https://arxiv.org/html/2610.40736#bib.bib16),[17](https://arxiv.org/html/2610.40736#bib.bib17)\]:

InDiscrPX​\(r\)≜limϵ→0\+\(PX​\(\(1\+ϵ\)​r\)−PX​\(r\)ϵ⋅FR​\(r\)\)=r⋅PX′​\(r\)PX​\(r\)=LIDPX​\(r\)\.\\begin\{split\}\\mathrm\{InDiscr\}\_\{P\_\{X\}\}\(r\)&\\triangleq\\lim\\limits\_\{\\epsilon\\to 0^\{\+\}\}\\Bigg\(\\frac\{P\_\{X\}\(\(1\+\\epsilon\)r\)\-P\_\{X\}\(r\)\}\{\\epsilon\\cdot F\_\{R\}\(r\)\}\\Bigg\)\\\\ &=\\frac\{r\\cdot P^\{\\prime\}\_\{X\}\(r\)\}\{P\_\{X\}\(r\)\}=\\mathrm\{LID\}\_\{P\_\{X\}\}\(r\)\.\\end\{split\}\(11\)Note that this definition of indiscriminability is unitless, does not require knowledge of the statistical parameters of the underlying distance distribution, and coincides precisely with local ID \(Eq\.[10](https://arxiv.org/html/2610.40736#S3.E10)\)\.

### 3\.3Estimation of Local Intrinsic Dimensionality

As the number of data elements increases, the distance from a query to itskk\-th nearest neighbor tends to decrease\. Asymptotically speaking, in the context of continuous distance distributions, the radius of the local vicinity within which a query result is determined tends to zero\. Accordingly, the discriminability of distances to the query should be assessed as the limitLIDFR∗\\mathrm\{LID\}^\{\*\}\_\{F\_\{R\}\}ofLIDFR​\(r\)\\mathrm\{LID\}\_\{F\_\{R\}\}\(r\)asrrtends to zero, whereFRF\_\{R\}is the Cumulative Distribution Function \(CDF\) of the random distance variableRR, representing distances from the query point to other data objects\. As it turns out, this limit can be estimated using Extreme Value Theory \(EVT\), which states that the tail of the distribution of the random distance variableRRcan be modeled as a Weibull distribution\. The Weibull distribution is specified by two parameters: the scaleσ\\sigmaand the shapeξ\\xi\. For a sufficiently small neighborhood distance threshold, the random distance variable representing the threshold excesses is Weibull\-distributed\[[15](https://arxiv.org/html/2610.40736#bib.bib15)\]\.

Concretely, the distances\(r1,…,rk\)\(r\_\{1\},\\ldots,r\_\{k\}\)to thekk\-nearest neighbors of a reference pointxxare instances of the random variableRR\. These instances can be used to estimate the Weibull shape parameterξ\\xi\. A maximum likelihood estimator \(MLE\) can be used to estimateξ\\xiand thereforeLIDFR∗\\mathrm\{LID\}^\{\*\}\_\{F\_\{R\}\}\[[15](https://arxiv.org/html/2610.40736#bib.bib15)\]:

LID^FR=−1ξ^=\(−1k∑i=1klnrirk\)−1\.\\mathrm\{\\widehat\{LID\}\}\_\{F\_\{R\}\}=\-\\frac\{1\}\{\\widehat\{\\xi\}\}=\\Bigg\(\-\\frac\{1\}\{k\}\\sum\_\{i=1\}^\{k\}\\ln\\frac\{r\_\{i\}\}\{r\_\{k\}\}\\Bigg\)^\{\-1\}\\\>\.\(12\)
Figure[1](https://arxiv.org/html/2610.40736#S3.F1)illustrates the model on three planar point sets whose distance distributions from the reference point satisfyFR​\(r\)=rmF\_\{R\}\(r\)=r^\{m\}exactly, form∈\{1,2,4\}m\\in\\\{1,2,4\\\}\. Starting from neighborhoods of equal size, an expansion of the radius by a common factor1\+ϵ1\+\\epsilonadmits comparatively few additional points when the local ID is low, and a great many when it is high; the corresponding expansion dimension recoversddin each case, as does the MLE\. The rightmost distance measure is thus the least discriminative, even though every case is embedded in the plane\.

Figure 1:LID as neighborhood growth\. Top: three point sets sampled so thatFR​\(r\)=rmF\_\{R\}\(r\)=r^\{m\}holds exactly form=1,2,4m=1,2,4, which is possible in the plane since LID depends on the distance distribution alone\. Each disc starts from a neighborhood of the same size,\|B⁡\(x,r\)\|≈120\|B\(x,r\)\|\\approx 120, and expands the radius by the same factor1\+ϵ=1\.41\+\\epsilon=1\.4: the expansion dimension over the two concentric balls returns the LID, so it is the relative gain in points, not the absolute gain, that reveals the dimension\. \(a\) On log\-log axes, the slope ofFRF\_\{R\}is the dimension\. \(b\) The\(1\+ϵ\)\(1\+\\epsilon\)ratio across distances, with the MLELID^FR\\widehat\{\\mathrm\{LID\}\}\_\{F\_\{R\}\}fromk=500k=500neighbors\.

## 4Method description

For a given reference point, LID quantifies the local rate of neighborhood growth, or equivalently, the difficulty of discriminating among nearby objects, with lower LID indicating greater local discriminability\. Because higher local discriminability benefits distance\-based learning, our selection strategy applies this principle in the transposed feature space, treating each feature as a point and evaluating it by the discriminability of its distances to neighboring features\. A feature’s overall quality is then given by the inverse of the LID estimate computed from itskk\-nearest\-neighbor distances\.

Algorithm 1LID Unsupervised Feature Selection0:Dataset

X∈ℝn×dX\\in\\mathbb\{R\}^\{n\\times d\}and feature\-neighborhood size

k∈\[2,d\)k\\in\[2,d\)\.

0:Feature\-importance score vector

SS\.

1:

X⊤←transpose⁡\(X\)X^\{\\top\}\\leftarrow\\operatorname\{transpose\}\(X\)
2:Remove duplicate rows of

X⊤X^\{\\top\}\.

3:For each

i∈\{1,…,d\}i\\in\\\{1,\\ldots,d\\\}, compute the

kknearest neighboring features of

fif\_\{i\}in

X⊤X^\{\\top\}, excluding

fif\_\{i\}itself\. Store the neighbor distances in

DD, where

Di​ℓD\_\{i\\ell\}is the distance from

fif\_\{i\}to its

ℓ\\ell\-th nearest neighbor, and store the corresponding feature indices in

II, where

Ii​ℓI\_\{i\\ell\}is the index of the

ℓ\\ell\-th nearest neighbor of

fif\_\{i\}\.

4:while

∃i:Di​1=Di​k\\exists\\,i:D\_\{i1\}=D\_\{ik\}do

5:

k←k\+1k\\leftarrow k\+1
6:Extend

DDand

IIwith the

kk\-th nearest neighbor of each feature\.

7:endwhile

8:Estimate the LID of each feature and compute its initial score:

S\(fi\)←1LID^\(fi;Di,:\)S\(f\_\{i\}\)\\leftarrow\\frac\{1\}\{\\widehat\{\\mathrm\{LID\}\}\(f\_\{i\};D\_\{i,:\}\)\},

i=1,…,di=1,\\ldots,d\.

9:Let

π=\(π1,π2,…,πd\)\\pi=\(\\pi\_\{1\},\\pi\_\{2\},\\ldots,\\pi\_\{d\}\)be a permutation of

\{1,…,d\}\\\{1,\\ldots,d\\\}such that

S⁡\(fπr\)≥S⁡\(fπs\)S\(f\_\{\\pi\_\{r\}\}\)\\geq S\(f\_\{\\pi\_\{s\}\}\)for all

r<sr<s\.

10:Initialize the selected\-index set

F←∅F\\leftarrow\\emptysetand the reserve\-index set

R←∅R\\leftarrow\\emptyset\.

11:for

r←1r\\leftarrow 1to

dddo

12:

i←πri\\leftarrow\\pi\_\{r\}
13:

𝑖𝑠​\_​𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟←false\\mathit\{is\\\_neighbor\}\\leftarrow\\mathrm\{false\}
14:for

j∈Fj\\in Fdo

15:if

i∈\{Ij​1,…,Ij​k\}i\\in\\\{I\_\{j1\},\\ldots,I\_\{jk\}\\\}then

16:

𝑖𝑠​\_​𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟←true\\mathit\{is\\\_neighbor\}\\leftarrow\\mathrm\{true\}
17:break

18:endif

19:endfor

20:if

𝑖𝑠​\_​𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟\\mathit\{is\\\_neighbor\}then

21:

R←R∪\{i\}R\\leftarrow R\\cup\\\{i\\\}
22:else

23:

F←F∪\{i\}F\\leftarrow F\\cup\\\{i\\\}
24:endif

25:endfor

26:if

R≠∅R\\neq\\emptysetthen

27:

Smax←maxi∈R⁡S⁡\(fi\)S\_\{\\max\}\\leftarrow\\max\_\{i\\in R\}S\(f\_\{i\}\)
28:for

i∈Ri\\in Rdo

29:

S⁡\(fi\)←−\(1−S⁡\(fi\)/Smax\)S\(f\_\{i\}\)\\leftarrow\-\(1\-S\(f\_\{i\}\)/S\_\{\\max\}\)
30:endfor

31:endif

32:return

S=\(S⁡\(f1\),…,S⁡\(fd\)\)S=\\bigl\(S\(f\_\{1\}\),\\ldots,S\(f\_\{d\}\)\\bigr\)\.

We detail our algorithm in the following; a sketch is provided in Alg\.[1](https://arxiv.org/html/2610.40736#alg1)\. LetX=\[f1,…,fd\]∈ℝn×dX=\[f\_\{1\},\\ldots,f\_\{d\}\]\\in\\mathbb\{R\}^\{n\\times d\}denote the input data matrix, wherennis the number of samples,ddis the number of features, and each column vectorfi∈ℝnf\_\{i\}\\in\\mathbb\{R\}^\{n\}contains the values of theii\-th feature across all samples\. Since the goal is to evaluate relationships among features rather than among samples, LID\-based feature selection first transposes the data matrix, yieldingX⊤∈ℝd×n\{X^\{\\top\}\\in\\mathbb\{R\}^\{d\\times n\}\}\. In this representation, each row corresponds to a feature described by its values across the dataset\. The algorithm can therefore construct neighborhoods in the transposed space, such that features with similar value profiles across the samples are considered close to one another\.

Given a user\-defined neighborhood sizekkand a chosen distance measure, the algorithm computes thekknearest neighboring features of each featurefif\_\{i\}\. The resulting neighbor distances are stored in a matrixDD, whereDi​ℓD\_\{i\\ell\}denotes the distance fromfif\_\{i\}to itsℓ\\ell\-th nearest neighbor\. The corresponding neighbor indices are stored in a matrixII, whereIi​ℓI\_\{i\\ell\}identifies the index of theℓ\\ell\-th nearest neighbor offif\_\{i\}\. The distances in theii\-th row ofDDare then used to estimate the Local Intrinsic Dimensionality offif\_\{i\}, denoted byLID^\(fi;Di,:\)\\widehat\{\\mathrm\{LID\}\}\(f\_\{i\};D\_\{i,:\}\)\. Before the neighbor search, duplicate features are removed, since they would result in zero neighbor distances\. Moreover, the LID estimate of a feature is undefined if all of itskkneighbor distances are equal; in this case,kkis increased until every feature has at least two distinct neighbor distances\.

A lower LID estimate indicates a lower relative rate of neighborhood growth and, according to the connection between LID and indiscriminability, greater local discriminability among neighboring features\. In the transposed space, this suggests that the feature has a comparatively distinctive value profile across the samples and is less locally redundant with nearby features\. Such a feature may therefore have greater potential to preserve distinctions among the points ofXX, although this implication is heuristic rather than guaranteed by the LID estimate alone\. The initial importance score of each feature is therefore defined as the inverse of its estimated LID:

S\(fi\)=1LID^\(fi;Di,:\)\.S\(f\_\{i\}\)=\\frac\{1\}\{\\widehat\{\\mathrm\{LID\}\}\(f\_\{i\};D\_\{i,:\}\)\}\.Accordingly, features with lower estimated LID values receive higher initial importance scores\.

After computing the initial scores, the features are sorted in descending order of importance\. The resulting ordered list is denoted by

Ω∗=\(f1∗,f2∗,…,fd∗\),\\Omega^\{\*\}=\(f^\{\*\}\_\{1\},f^\{\*\}\_\{2\},\\ldots,f^\{\*\}\_\{d\}\),wherefi∗f^\{\*\}\_\{i\}is the feature at rankii, andS⁡\(fi∗\)≥S⁡\(fj∗\)S\(f^\{\*\}\_\{i\}\)\\geq S\(f^\{\*\}\_\{j\}\)for alli<ji<j\. The LID\-based feature selection algorithm then performs a refinement step to reduce redundancy among highly ranked features\. Two sets are initialized: the selected\-feature setFF, which stores representative features, and the reserve setRR, which stores features considered locally redundant with respect to previously selected features\.

The algorithm scans the ranked listΩ∗\\Omega^\{\*\}from the highest\-scoring feature to the lowest\-scoring feature\. For each candidate featurefi∗f^\{\*\}\_\{i\}, it determines whetherfi∗f^\{\*\}\_\{i\}belongs to thekk\-neighborhood of any feature already included inFF\. If it does not belong to the neighborhood of any selected feature, it is added toFFand treated as non\-redundant under the neighborhood criterion\. Otherwise, it is added toRR, since it is locally similar to at least one previously selected feature\.

Finally, the scores of the features stored in the reserve setRRare adjusted\. IfR≠∅R\\neq\\varnothing, let

Smax=maxf∈R⁡S⁡\(f\)S\_\{\\max\}=\\max\_\{f\\in R\}S\(f\)denote the largest initial importance score among the reserved features\. For each featurefr∈Rf\_\{r\}\\in R, the score is updated according to

S⁡\(fr\)=−\(1−S⁡\(fr\)Smax\)\.S\(f\_\{r\}\)=\-\\left\(1\-\\frac\{S\(f\_\{r\}\)\}\{S\_\{\\max\}\}\\right\)\.Since the initial scores are positive andS⁡\(fr\)≤SmaxS\(f\_\{r\}\)\\leq S\_\{\\max\}, this transformation assigns a score in the interval\(−1,0\]\(\-1,0\]to every reserved feature while preserving their relative ordering\. In particular, the highest\-scoring feature inRRreceives a refined score of00, whereas the remaining reserved features receive negative scores\. Features in the selected setFFretain their positive initial scores and therefore precede all features inRRin the final ranking\.

As a result of this adjustment, a complete ranking is produced in which locally non\-redundant features are prioritized, while the relative ordering of the reserved features is retained\. Consequently, a user may select any desired number of top\-ranked features without rerunning the feature\-selection procedure\. IfR=∅R=\\varnothing, no score adjustment is required\. The final output of LIDFS is the feature\-importance vector

S=\(S⁡\(f1\),S⁡\(f2\),…,S⁡\(fd\)\),S=\\bigl\(S\(f\_\{1\}\),S\(f\_\{2\}\),\\ldots,S\(f\_\{d\}\)\\bigr\),\(13\)which can be used to rank the features for unsupervised feature selection\.

LID\-based feature\-importance scores quantify the local discriminability of features in the transposed feature space\. A feature with a lower estimated LID has a more distinctive value profile relative to its neighboring features and is therefore treated as a potentially informative representative feature\. The refinement step further prioritizes features from different neighborhoods, reducing the selection of locally redundant features\. The neighborhood sizekkcontrols the locality of both the LID estimates and the redundancy criterion: smaller values ofkkemphasize fine\-scale neighborhood structure but yield less stable estimates, whereas larger values yield smoother estimates but may merge distinct local structures\. The choice ofkkthus involves a trade\-off between locality and estimation stability\.

#### Computational complexity

The time complexity of LIDFS is analyzed for a dataset withnnsamples andddfeatures, assuming a distance measure that is computed in𝒪⁡\(n\)\\mathcal\{O\}\(n\)time for a pair of features, such as the Euclidean distance\. Since the removal of duplicate features can only reduce the number of features,ddalso bounds the number of features that remain after this step\. Transposing the data matrix requires𝒪⁡\(n​d\)\\mathcal\{O\}\(nd\)time, and duplicate features are removed in𝒪⁡\(n​d\)\\mathcal\{O\}\(nd\)expected time by hashing the rows ofX⊤X^\{\\top\}\. The dominant cost is thekk\-nearest\-neighbor search in the transposed space\. An exact search computes the distances between all pairs of features in𝒪⁡\(d2​n\)\\mathcal\{O\}\(d^\{2\}n\)time, after which thekknearest neighbors of each feature are obtained by partial selection in𝒪⁡\(d\)\\mathcal\{O\}\(d\)time and sorted in𝒪⁡\(k​log⁡k\)\\mathcal\{O\}\(k\\log k\)time, resulting in𝒪⁡\(d2​n\+d​k​log⁡k\)\\mathcal\{O\}\(d^\{2\}n\+dk\\log k\)time in total\. The loop that increaseskkis only entered if allkkneighbor distances of a feature are equal, in which case the neighbor lists are extended from the already computed distances; even ifkkgrows tod−1d\-1, this requires at most𝒪⁡\(d2​log⁡d\)\\mathcal\{O\}\(d^\{2\}\\log d\)time for sorting all rows of the distance matrix\. The maximum\-likelihood estimate of LID \(Eq\.[12](https://arxiv.org/html/2610.40736#S3.E12)\) requires a single pass over thekkneighbor distances of a feature, and therefore the initial scores of all features are computed in𝒪⁡\(d​k\)\\mathcal\{O\}\(dk\)time\. Sorting the features by their scores requires𝒪⁡\(d​log⁡d\)\\mathcal\{O\}\(d\\log d\)time\. In the refinement step, the membership test is implemented with a Boolean array that marks the neighbors of all selected features, such that each candidate is checked in𝒪⁡\(1\)\\mathcal\{O\}\(1\)time and each insertion intoFFrequires𝒪⁡\(k\)\\mathcal\{O\}\(k\)time, resulting in𝒪⁡\(d​k\)\\mathcal\{O\}\(dk\)time for the entire scan\.222A direct implementation that scansFFfor every candidate requires𝒪⁡\(d2​k\)\\mathcal\{O\}\(d^\{2\}k\)time in the worst case, which does not exceed𝒪⁡\(d2​n\)\\mathcal\{O\}\(d^\{2\}n\)fork≤nk\\leq n\.Finally, the adjustment of the scores of the reserved features requires𝒪⁡\(\|R\|\)⊆𝒪⁡\(d\)\\mathcal\{O\}\(\|R\|\)\\subseteq\\mathcal\{O\}\(d\)time\. Consequently, the overall time complexity of LIDFS is

𝒪⁡\(d2​n\+d2​log⁡d\+d​k​log⁡k\)=𝒪⁡\(d2​n\),\\mathcal\{O\}\\bigl\(d^\{2\}n\+d^\{2\}\\log d\+dk\\log k\\bigr\)=\\mathcal\{O\}\(d^\{2\}n\),where the simplification holds forlog⁡d≤n\\log d\\leq n, which is satisfied by any practical dataset\. The running time of LIDFS is therefore quadratic in the number of features and linear in the number of samples, while the neighborhood sizekkonly affects lower\-order terms\. The space complexity is𝒪⁡\(n​d\+d2\)\\mathcal\{O\}\(nd\+d^\{2\}\)if the full distance matrix is stored, which reduces to𝒪⁡\(n​d\+d​k\)\\mathcal\{O\}\(nd\+dk\)if the distances are computed in blocks and only the nearest neighbors of each feature are retained\. For datasets with a very large number of features, the exact neighbor search can be replaced by an approximate nearest\-neighbor search to alleviate the quadratic dependence ondd\.

## 5Experimental framework

In this section, we describe the experimental setup used to compare the feature selection performance of LID\-based Feature Selection \(LIDFS\) with various unsupervised feature selection methods\. Competitors include traditional methods \(Variance\-based\[[40](https://arxiv.org/html/2610.40736#bib.bib40)\], Correlation\-based\[[41](https://arxiv.org/html/2610.40736#bib.bib41),[42](https://arxiv.org/html/2610.40736#bib.bib42)\], Laplacian Score\[[13](https://arxiv.org/html/2610.40736#bib.bib13)\], and MCFS\[[14](https://arxiv.org/html/2610.40736#bib.bib14)\]\) as well as recent methods \(SCFS\[[25](https://arxiv.org/html/2610.40736#bib.bib25)\], SOGFS\[[23](https://arxiv.org/html/2610.40736#bib.bib23)\], LLSRFS\[[24](https://arxiv.org/html/2610.40736#bib.bib24)\], and VCSDFS\[[26](https://arxiv.org/html/2610.40736#bib.bib26)\]\)\.

### 5\.1Evaluation Metrics

To evaluate the feature selection methods across both supervised and unsupervised downstream tasks, four key metrics are employed: Accuracy \(ACC\), Area Under the Curve \(AUC\), Clustering Accuracy \(CLSACC\), and Normalized Mutual Information \(NMI\)\.

#### Accuracy \(ACC\)

Accuracy is a fundamental metric for supervised classification tasks, representing the proportion of correct predictions\. For a set ofnninstances with true labelslil\_\{i\}and predicted labelsl^i\\hat\{l\}\_\{i\}, it is calculated as:

ACC=1n​∑i=1n𝕀⁡\(li=l^i\)\\text\{ACC\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\mathbb\{I\}\(l\_\{i\}=\\hat\{l\}\_\{i\}\)\(14\)where𝕀⁡\(⋅\)\\mathbb\{I\}\(\\cdot\)is the indicator function\.

#### Area Under the Curve \(AUC\)

AUC measures the ability of a classifier to distinguish between classes\. It is the area under the Receiver Operating Characteristic \(ROC\) curve, which plots the True Positive Rate \(Sensitivity\) against the False Positive Rate \(1 \- Specificity\) at various threshold settings\. AUC ranges from 0 to 1, with0\.50\.5indicating random performance and a higher value indicating better performance\. For a given threshold, withTP,FP,TN, andFNdenoting the numbers of true positives, false positives, true negatives, and false negatives, the True Positive Rate \(TPR\) and False Positive Rate \(FPR\) are defined as:

TPR=TPTP\+FNandFPR=FPFP\+TN\\text\{TPR\}=\\frac\{\\text\{TP\}\}\{\\text\{TP\}\+\\text\{FN\}\}\\quad\\text\{and\}\\quad\\text\{FPR\}=\\frac\{\\text\{FP\}\}\{\\text\{FP\}\+\\text\{TN\}\}\(15\)

#### Clustering Accuracy \(CLSACC\)

Clustering Accuracy assesses how well the clusters found by the algorithm match the true labels of the data\. It requires finding the best one\-to\-one mapping \(permutationπ\\pi\) between the cluster assignments \(cic\_\{i\}\) and the true labels \(lil\_\{i\}\)\.

CLSACC=1n​∑i=1n𝕀⁡\(li=π⁡\(ci\)\)\\text\{CLSACC\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\mathbb\{I\}\(l\_\{i\}=\\pi\(c\_\{i\}\)\)\(16\)wherennis the number of data points\. The optimal mappingπ\\piis typically found using the Hungarian algorithm\[[43](https://arxiv.org/html/2610.40736#bib.bib43)\]\.

#### Normalized Mutual Information \(NMI\)

NMI quantifies the degree of dependence between the clustering result \(CC\) and the true labels \(LL\)\. It is a normalization of the Mutual Information \(MI\), making it comparable across different cluster numbers\. NMI is calculated as:

NMI​\(C,L\)=2​MI​\(C,L\)H⁡\(C\)\+H⁡\(L\)\\text\{NMI\}\(C,L\)=\\frac\{2\\,\\text\{MI\}\(C,L\)\}\{H\(C\)\+H\(L\)\}\(17\)whereMI​\(C,L\)\\text\{MI\}\(C,L\)is the Mutual Information betweenCCandLL, andH⁡\(C\)H\(C\)andH⁡\(L\)H\(L\)are the entropies ofCCandLL, respectively\. The Mutual Information is given by:

MI​\(C,L\)=∑c∈C∑l∈LP⁡\(c,l\)​log⁡\(P⁡\(c,l\)P⁡\(c\)​P​\(l\)\)\\text\{MI\}\(C,L\)=\\sum\_\{c\\in C\}\\sum\_\{l\\in L\}P\(c,l\)\\log\\left\(\\frac\{P\(c,l\)\}\{P\(c\)P\(l\)\}\\right\)\(18\)

#### Feature Selection Dynamic Evaluation Metric \(FSDEM\)

In order to evaluate the overall process of feature selection, we use FSDEM combined with the aforementioned metrics\. FSDEM is calculated as:

FSDEM=∫abg⁡\(x\)​𝑑xb−a,\\text\{FSDEM\}=\\frac\{\\int\_\{a\}^\{b\}g\(x\)\\,dx\}\{b\-a\},\(19\)whereg⁡\(x\)g\(x\)indicates the approximated function based on different observations of an arbitrary evaluation metric, andaaandbbindicate the specific range of values \(percentage of selected features\) in which the performance is evaluated\.

### 5\.2Datasets

We conduct experiments on 23 high\-dimensional datasets from the scikit\-feature repository\[[44](https://arxiv.org/html/2610.40736#bib.bib44)\]\. Table[1](https://arxiv.org/html/2610.40736#S5.T1)presents an overview of the characteristics of the datasets\. The datasets vary in the number of instances, features, and domains\. This repository is the standard source for feature selection benchmarks\.

Table 1:Characteristics of the high\-dimensional benchmark datasets\.IDDatasetInstancesFeaturesD1ALLAML727,129D2arcene20010,000D3Carcinom1749,182D4CLL\_SUB\_11111111,340D5COIL201,4401,024D6colon622,000D7gisette7,0005,000D8GLI\_858522,283D9GLIOMA504,434D10Isolet1,560617D11leukemia727,070D12lung2033,312D13lung\_discrete73325D14madelon2,600500D15ORL4001,024D16orlraws10P10010,304D17pixraw10P10010,000D18Prostate\_GE1025,966D19SMK\_CAN\_18718719,993D20TOX\_1711715,748D21warpAR10P1302,400D22warpPIE10P2102,420D23Yale1651,024
### 5\.3Evaluation Configuration

We perform our experiments using theFSEVALbenchmarking suite\[[45](https://arxiv.org/html/2610.40736#bib.bib45)\], a specialized framework designed for the extensive and comprehensive evaluation of feature selection algorithms\. The supervised evaluation is conducted with five downstream classifiers, namely KNN\[[46](https://arxiv.org/html/2610.40736#bib.bib46)\], Decision Tree\[[47](https://arxiv.org/html/2610.40736#bib.bib47)\], SVM\[[48](https://arxiv.org/html/2610.40736#bib.bib48)\], Random Forest\[[49](https://arxiv.org/html/2610.40736#bib.bib49)\], and XGBoost\[[50](https://arxiv.org/html/2610.40736#bib.bib50)\]with a 5\-fold stratified cross\-validation to provide a robust evaluation\. For brevity, we present the detailed results for the Random Forest classifier in Section[6](https://arxiv.org/html/2610.40736#S6)\. The results for the other classifiers \(reported in the Appendix\) lead to the same conclusions\. Multiclass AUC is macro\-averaged\. For CLSACC and NMI, we use the average of 10 runs of thekk\-means algorithm\.kk\-means is always given the true number of classes\. For the random baseline, the evaluation is repeated 100 times to clearly reflect the standard deviation\. The average value is used for further analysis\. For LIDFS, we usek=5k=5as a default setting to calculate LID values for each feature\. We favor smaller values forkk, as fine\-grained neighborhoods are deemed more important than the approximation accuracy of the LID\. We use Euclidean distance as the chosen distance measure and we also include Random feature selection\[[51](https://arxiv.org/html/2610.40736#bib.bib51)\]as a baseline for our evaluation\. Runtime tests ran on Ubuntu with an AMD Ryzen 9 9950X \(128 GB RAM\)\.

We conduct two different experiments to evaluate the performance of the feature selection process\. First, we evaluate the overall feature selection process by selecting from 5% to 100% of the features with a step size of 5%\. As the datasets are very high\-dimensional, we also conduct a different experiment by selecting from 0\.5% to 10% of features with a step size of 0\.5%\. This allows us to have a more realistic view of the performance of feature selection algorithms to make sure the high number of selected features relative to the number of instances does not obscure the results\.

## 6Results

Figure 2:Runtime analysis of the feature selection methods\. Theyy\-axis is presented in logarithmic scale\.Figure 3:Comparison of the feature selection performance of the unsupervised feature selection methods on the Isolet dataset with the random baseline over the full range of the number of selected features\.In this section, we present the experimental results based on the described evaluation framework\. We aim to evaluate different aspects of unsupervised feature selection algorithms, including efficiency and performance, to comprehensively assess the characteristics of LIDFS against the competitors\.

### 6\.1Runtime Analysis

Our first experiment evaluates the runtime of the unsupervised feature selection algorithms\. Demonstrating computational efficiency is essential to understanding the practical significance of the proposed method’s performance\. This experiment considers two variables: the number of instances and the number of features\. We report two settings, in each of which one variable is fixed at 100 while the other varies from 1000 to 20000 in steps of 500\. Runtime is capped at one hour per method; the first time a method exceeds this cap, the elapsed time is recorded and no further settings are evaluated for that method\. Figure[2](https://arxiv.org/html/2610.40736#S6.F2)summarizes the results, with theyy\-axis on a logarithmic scale for readability\.

As expected, the random baseline is by far the most efficient method, as it involves no significant computation\. It is followed by traditional methods such as variance\-based and correlation\-based feature selection\. Recent state\-of\-the\-art methods are computationally more expensive\. LIDFS shows competitive results in the experiment with varying numbers of features and highly efficient performance when the number of instances is varied\. The only recent method that consistently achieves better efficiency is VCSDFS, which outperforms LIDFS by a small margin\. SOGFS and LLSRFS are the most expensive methods, and their runtime is capped at an early stage\. As we aim to provide consistent and comprehensive experiments on large, high\-dimensional datasets, we omit SOGFS and LLSRFS from the remainder of the experiments\.

Figure 4:Comparison of the performance of the unsupervised feature selection methods on the Isolet dataset with the random baseline over the first 10% of the features\.
### 6\.2Performance Analysis

We analyze the performance of the proposed method in comparison with traditional and state\-of\-the\-art unsupervised feature selection algorithms based on the 23 benchmark datasets and the four different evaluation metrics\.

Figure[3](https://arxiv.org/html/2610.40736#S6.F3)presents an example of the four evaluation metrics on the Isolet dataset for the full\-range experiment\. LIDFS achieves the top performance while also maintaining the efficiency demonstrated earlier, and it generally outperforms the random baseline\. Many feature selection methods, including recent state\-of\-the\-art algorithms, fail to outperform the random baseline\. This might seem surprising, but is in fact consistent with previous observations\[[51](https://arxiv.org/html/2610.40736#bib.bib51)\]\. The same pattern holds for the second case, in which the range of the number of selected features is limited to 10%, as presented in Fig\.[4](https://arxiv.org/html/2610.40736#S6.F4)\.

An interesting observation in Fig\.[3](https://arxiv.org/html/2610.40736#S6.F3)for the full\-range experiment is that, for the top\-performing methods, the performance curve flattens after roughly the top15%15\\%of features \(as ranked by each method\) have been selected\. Naïve, low\-performing methods such as correlation\-based feature selection continue to improve up to around the80%80\\%selection mark, or beyond\. This is consistent with our assumption in LIDFS: within each feature neighborhood, one feature is designated as the representative and the rest are moved to the reserve set, reflecting their possible redundancy\. The results suggest that, with an appropriate choice ofkkfor the neighborhood search, LIDFS identifies this redundancy, assigning positive importance values only to features whose selection is expected to increase the informational value of the lower\-dimensional representation based on LIDFS’s perspective\. Figure[4](https://arxiv.org/html/2610.40736#S6.F4)confirms this, as the performance curves remain almost monotonically increasing throughout the10%10\\%selection experiment\.

Feature selection performance varies across datasets; we selected one exhibiting a commonly observed pattern for illustration\. Individual results for all datasets are available in the Appendix\.\. The FSDEM score, which shows the overall performance over a range, is presented for both experiments in Figures[5](https://arxiv.org/html/2610.40736#S6.F5)and[7](https://arxiv.org/html/2610.40736#S6.F7)\. We observe that LIDFS constitutes the frontier almost consistently, providing a reliable and efficient solution for unsupervised feature selection\. To analyze the overall performance across all benchmark datasets, we use critical difference diagrams, which rank methods by performance from right to left, following the methodology of Demšar\[[52](https://arxiv.org/html/2610.40736#bib.bib52)\]\.

Figure 5:Mean AUC, ACC, NMI, and CLSACC \(FSDEM\) for the full \(100%\) selection scenarios\. LIDFS constitutes the frontier almost consistently\.Figure 6:Critical difference diagram over the full range of number of features for the four different metrics based on the average performance measured by the FSDEM score\.The critical difference diagrams for the different metrics, over the full range of feature numbers and the10%10\\%range, are shown in Fig\.[6](https://arxiv.org/html/2610.40736#S6.F6)and Fig\.[8](https://arxiv.org/html/2610.40736#S6.F8), respectively\. This analysis uses the FSDEM score\[[3](https://arxiv.org/html/2610.40736#bib.bib3)\], computed as the average performance over the range for each of the four metrics\.

Figure 7:Mean AUC, ACC, NMI, and CLSACC \(FSDEM\) for the aggressive \(10%\) selection scenarios\. LIDFS constitutes the frontier almost consistently\.Figure 8:Critical difference diagram over the first 10% of number of features for the four different metrics based on the average performance measured by the FSDEM score\.LIDFS is in most cases the highest\-performing method for unsupervised feature selection\. For supervised downstream tasks, LIDFS, SCFS\[[25](https://arxiv.org/html/2610.40736#bib.bib25)\], and MCFS\[[14](https://arxiv.org/html/2610.40736#bib.bib14)\]form the top statistical group, with LIDFS ranked first and a clear margin over the remaining methods\. Other state\-of\-the\-art methods, such as VCSDFS\[[26](https://arxiv.org/html/2610.40736#bib.bib26)\], perform even worse than the random baseline\. For the unsupervised evaluation, LIDFS is either the best or the second\-best \(in a tie\), without a statistically significant difference or a large margin relative to the best method\. On the efficiency frontier, LIDFS shows a lower computational complexity and runtime compared to the closest competitor, SCFS\[[25](https://arxiv.org/html/2610.40736#bib.bib25)\]\. It is noteworthy that LIDFS works in a truly unsupervised setting, while some of the competitors, including SCFS\[[25](https://arxiv.org/html/2610.40736#bib.bib25)\]and MCFS\[[14](https://arxiv.org/html/2610.40736#bib.bib14)\], use information extracted from the labels, such as the number of unique classes\. The results for the other classifiers are almost identical to those for the Random Forest classifier and lead to the same conclusions\.

### 6\.3Sensitivity Analysis

The neighborhood sizekkused to estimate the LID of each point is the central hyperparameter of the proposed feature selection procedure\. As discussed earlier, theoretically, larger values ofkkare expected to produce more accurate LID estimates, since the estimator averages over a larger sample\. At the same time, a largerkkenlarges the neighborhood over which LID is computed possibly beyond local characteristics, reducing the granularity of the discrimination between features\. These two effects act in opposite directions, so the net effect ofkkon downstream feature selection quality is not obvious a priori and must be assessed empirically\. We evaluatek∈\{5,10,25,50,100\}k\\in\\\{5,10,25,50,100\\\}on all 23 benchmark datasets under the two feature\-selection scenarios, retaining up to 10% of the original features, and full selection up to 100%\.

#### Downstream performance

Figure[9](https://arxiv.org/html/2610.40736#S6.F9)summarizes different evaluation metrics as a function ofkk, averaged over datasets and over the feature\-selection fraction within each scenario\. This figure shows the overall trend of different performance metrics whenkkincreases, which is generally decreasing\. This illustrates that the granularity of neighborhoods in the feature selection process is more important than the smoothness of the LID estimation\.

Figure 9:Mean AUC, ACC, NMI, and CLSACC as a function of the neighborhood sizekk, averaged over datasets and over the feature\-selection fraction, for the aggressive \(10%\) and full \(100%\) selection scenarios\. Error bars show±1\\pm 1standard error of the mean over the 23 datasets\. For all metrics higher values are better\.
#### Per\-dataset heterogeneity and practical guidance

The FSDEM score based on each metric on different datasets is presented, for the full selection and aggressive selection scenarios, in Figure[10](https://arxiv.org/html/2610.40736#S6.F10)and Figure[11](https://arxiv.org/html/2610.40736#S6.F11), respectively\. These figures highlight that smaller values ofkk, which yield finer neighborhoods, generally outperform larger values, which yield more stable LID estimates\. However, it is evident that no single value ofkkis optimal on every dataset\. The rank heatmap in Figure[12](https://arxiv.org/html/2610.40736#S6.F12)shows that the preference for smallkkis strongest for AUC and ACC, while NMI and CLSACC are comparatively indifferent\. In the full \(100%\) selection scenario, the best FSDEM score averaged over the four metrics is attained atk=5k=5on 7 of the 23 datasets, atk=10k=10on 6, atk=25k=25on 1, atk=50k=50on 5, and atk=100k=100on 4\. Small\-to\-moderate neighborhoods \(k=5k=5ork=10k=10\) are therefore individually optimal on 13 of 23 datasets \(57%\), consistent with the aggregate trend, but the remaining 43% of datasets favork≥25k\\geq 25, so dataset\-specific tuning ofkkremains worthwhile when computational budget allows\. In the absence of tuning, our results support using a smallkk\(k=5k=5as used and discussed in the main experimental results and discussions\) rather than a large neighborhood\.

Figure 10:The performance of LIDFS with different choices ofkkon different datasets and metrics for the full \(100%\) selection scenario\. Smaller values ofkkshow a better overall performance\.Figure 11:The performance of LIDFS with different choices ofkkon different datasets and metrics for the aggressive \(10%\) selection scenario\. Smaller values ofkkshow a better overall performance\.![Refer to caption](https://arxiv.org/html/2610.40736v1/fig_rank_heatmap.png)
Figure 12:Average ranks across the metrics for both scenarios\. The preference for small neighborhoods is strongest for AUC and ACC; NMI and CLSACC are comparatively flat\.
#### Summary

The sensitivity analysis was conducted with all classifiers and shows the same trend as for the Random Forest classifier, where small values ofkk, particularlyk=5k=5andk=10k=10, perform comparably, and larger values lead to a slight degradation in performance\. Taken together, the results confirm the theoretical trade\-off motivating this analysis: largerkkis expected to improve the fidelity of the LID estimate itself, yet this improvement does not transfer into better downstream performance; on average the coarser local granularity mildly degrades the four metrics we track\. A small\-to\-moderate neighborhood size is therefore the recommended default, with per\-dataset tuning ofkkavailable as an option when the observed heterogeneity across datasets justifies the additional cost\.

## 7Conclusion

We proposed a novel unsupervised feature selection algorithm based on LID to compute feature importance values and select informative features without requiring labeled data\. We theoretically motivated LID and its suitability for the unsupervised feature selection task, and provided a brief but comprehensive review of progress in the field\. We selected a large number of datasets to comprehensively and thoroughly evaluate the proposed method against classic and state\-of\-the\-art algorithms, including a random baseline to further guide the evaluation\. Finally, we conducted an in\-depth runtime analysis to compare the computational efficiency of the different unsupervised feature selection algorithms considered, and a sensitivity analysis of the neighborhood sizekk\.

The experimental results show that the proposed unsupervised feature selection method, LIDFS, offers the best overall observed performance against all competitors, outperforming competitors in most cases and remaining highly competitive in the rest\. The superiority of LIDFS, while not statistically significant, is remarkable in particular as LIDFS shows a better computational complexity in comparison with the closest competitors \(e\.g\., SCFS\)\. These observations highlight the effectiveness and efficiency of LIDFS in conducting a truly unsupervised feature selection with high performance figures and low computational complexity, without using limited or partial information from the labels like some competitors \(e\.g\., SCFS, MCFS\) do\.

### 7\.1Discussion

LIDFS differs from most other methods in the literature in the sense that it uses neither information\-theoretic nor spectral measures to select features\. Instead, it directly assesses the discriminability of features via the estimation of LID\. One major advantage of LID is that, aside from continuity, it requires no knowledge or assumptions concerning the distribution of the data\. The main drawback is that LIDFS requires a relatively smooth distribution of distance values, which is more likely to occur when the data are dense, high\-dimensional, or both\. Nevertheless, such settings are those in which classical feature selection methods often fail, either due to high computational complexity or unstable performance, sometimes falling even below the random baseline\.

The underlying logic of LIDFS is to assign greater importance, via an inverse relationship with the LID estimate, to features that reflect a lower\-dimensional local structure relative to others, and therefore exhibit higher discriminability\. It also includes a pruning step that prevents the selection of too many mutually close features, maintaining low redundancy in the selection process\. As shown in the experimental results, this discrimination between features provides a systematic approach to identifying redundancy, an ability tied to the neighborhood sizekkused when computing LID\. This orientation toward low\-dimensional local structure is reflected in LIDFS’s strong performance relative to the other algorithms in the experiments\.

### 7\.2Future work

As data volume increases, the cost of building an exact index becomes prohibitive, motivating the use of approximate nearest\-neighbor algorithms such asϵ\\epsilon\-NNS\[[2](https://arxiv.org/html/2610.40736#bib.bib2)\]or NN\-Descent\[[53](https://arxiv.org/html/2610.40736#bib.bib53)\]in place of exact search\. Using approximate nearest\-neighbor distances may adversely affect LID estimation and, consequently, alter the feature ranking; however, trading off feature quality for computation time can be advantageous when the dataset is large or high\-dimensional\.

One possible extension of this approach is to consider other models of intrinsic dimensionality as an alternative to LID for assessing feature quality\. Although such models may lack the theoretical guarantees regarding the discriminability of distance measures, or may have higher computational complexity or greater sensitivity to noise\[[15](https://arxiv.org/html/2610.40736#bib.bib15)\], they may still merit consideration, particularly if they prove more reliable for non\-smooth distance distributions or for smaller data samples\.

There are several other possible directions for future work on ID\-based feature selection\. First, it would be interesting to consider the problem of unsupervised tuning of the parameters of ID\-based feature selection\. Also, our approach to unsupervised feature ranking could be extended to supervised applications by restricting the ID estimation to training examples with common labels\. Finally, the algorithmic design of LIDFS could be revisited to yield a more granular feature\-importance calculation, either through a refinement step that avoids assigning similar importance values within the same neighborhood, or through other approaches independent ofkk\. This, however, must be done with careful consideration to maintain the low computational overhead of the current unsupervised feature selection process\.

## Acknowledgments

This study was funded by the Innovation Fund Denmark project “PREPARE: Personalized Risk Estimation and Prevention of Cardiovascular Disease”\.

## References

- \[1\]I\. Guyon, A\. Elisseeff, An introduction to variable and feature selection, JMLR 3 \(2003\)\.
- \[2\]P\. Indyk, R\. Motwani, Approximate nearest neighbors: towards removing the curse of dimensionality, in: STOC, ACM, 1998\.
- \[3\]M\. Rajabinasab, A\. D\. Lautrup, T\. Hyrup, A\. Zimek, A dynamic evaluation metric for feature selection, in: Similarity Search and Applications, Springer Nature Switzerland, Cham, 2025\.
- \[4\]F\. Camastra, Data dimensionality estimation methods: a survey, Pattern Recognition 36 \(12\) \(2003\)\.
- \[5\]S\. Niijima, Y\. Okuno, Laplacian linear discriminant analysis approach to unsupervised feature selection, TCBB 6 \(4\) \(2009\)\.
- \[6\]Y\. Saeys, I\. Inza, P\. Larrañaga, A review of feature selection techniques in bioinformatics, Bioinformatics 23 \(19\) \(2007\)\.
- \[7\]A\.\-C\. Haury, P\. Gestraud, J\.\-P\. Vert, The influence of feature selection methods on accuracy, stability and interpretability of molecular signatures, PloS One 6 \(12\) \(2011\)\.
- \[8\]J\. Feng, H\. Xu, S\. Yan, Robust PCA in high\-dimension: A deterministic approach, in: ICML, 2012\.
- \[9\]R\. Wald, T\. Khoshgoftaar, A\. Napolitano, The importance of performance metrics within wrapper feature selection, in: IRI, IEEE, 2013\.
- \[10\]X\. He, M\. Ji, C\. Zhang, H\. Bao, A variance minimization criterion to feature selection using laplacian regularization, PAMI 33 \(10\) \(2011\)\.
- \[11\]B\. Wang, A\. Goldenberg, Gradient\-based laplacian feature selection, CoRR abs/1404\.2948 \(2014\)\.
- \[12\]Z\. Zhao, H\. Liu, Spectral feature selection for supervised and unsupervised learning, in: ICML, ACM, 2007\.
- \[13\]X\. He, D\. Cai, P\. Niyogi, Laplacian score for feature selection, in: NIPS, 2005\.
- \[14\]D\. Cai, C\. Zhang, X\. He, Unsupervised feature selection for multi\-cluster data, in: KDD, ACM, 2010\.
- \[15\]L\. Amsaleg, O\. Chelly, T\. Furon, S\. Girard, M\. E\. Houle, K\. Kawarabayashi, M\. Nett, Estimating local intrinsic dimensionality, in: KDD, ACM, 2015\.
- \[16\]M\. E\. Houle, Dimensionality, Discriminability, Density & Distance Distributions, in: ICDMW, IEEE, 2013\.
- \[17\]M\. E\. Houle, Local intrinsic dimensionality I: An extreme\-value\-theoretic foundation for similarity applications, in: SISAP, 2017\.
- \[18\]H\. Peng, F\. Long, C\. Ding, Feature selection based on mutual information criteria of max\-dependency, max\-relevance, and min\-redundancy, PAMI 27 \(8\) \(2005\)\.
- \[19\]T\. M\. Cover, J\. A\. Thomas, Elements of Information Theory, 1991\.
- \[20\]M\. A\. Hall, L\. A\. Smith, Feature selection for machine learning: Comparing a correlation\-based filter approach to the wrapper\., in: FLAIRS Conference, Vol\. 1999, 1999\.
- \[21\]H\. Liu, R\. Setiono, Chi2: Feature selection and discretization of numeric attributes, in: TAI, IEEE, 1995\.
- \[22\]R\. O\. Duda, P\. E\. Hart, D\. G\. Stork, Pattern Classification, 2001\.
- \[23\]F\. Nie, W\. Zhu, X\. Li, Unsupervised feature selection with structured graph optimization, in: AAAI, AAAI Press, 2016, pp\. 1302–1308\.
- \[24\]C\. Wang, J\. Wang, Z\. Gu, J\.\-M\. Wei, J\. Liu, Unsupervised feature selection by learning exponential weights, Pattern Recognition 148 \(2024\)\.[doi:https://doi\.org/10\.1016/j\.patcog\.2023\.110183](https://doi.org/https://doi.org/10.1016/j.patcog.2023.110183)\.
- \[25\]M\. G\. Parsa, H\. Zare, M\. Ghatee, Unsupervised feature selection based on adaptive similarity learning and subspace clustering, Eng\. Appl\. Artif\. Intell\. 95 \(2020\)\.[doi:10\.1016/J\.ENGAPPAI\.2020\.103855](https://doi.org/10.1016/J.ENGAPPAI.2020.103855)\.
- \[26\]S\. Karami, F\. Saberi\-Movahed, P\. Tiwari, P\. Marttinen, S\. Vahdati, Unsupervised feature selection based on variance\-covariance subspace distance, Neural Networks 166 \(2023\)\.[doi:10\.1016/J\.NEUNET\.2023\.06\.018](https://doi.org/10.1016/J.NEUNET.2023.06.018)\.
- \[27\]X\. Zhang, R\. C\. de Amorim, Scalable unsupervised feature selection via weight stability \(2025\)\.[arXiv:2506\.06114](http://arxiv.org/abs/2506.06114)\.
- \[28\]K\. Jiang, T\. Cao, L\. Zhu, Q\. Sun, Adaptive and flexibleℓ1\\ell\_\{1\}\-norm graph embedding for unsupervised feature selection, Applied Intelligence 54 \(22\) \(2024\)\.[doi:10\.1007/s10489\-024\-05760\-z](https://doi.org/10.1007/s10489-024-05760-z)\.
- \[29\]Z\. Cao, X\. Xie, Y\. Li, Multi\-view unsupervised feature selection with consensus partition and diverse graph, Information Sciences 661 \(2024\)\.
- \[30\]J\.\-S\. Wu, Y\. Li, J\.\-X\. Gong, W\. Min, Collaborative and discriminative subspace learning for unsupervised multi\-view feature selection, Engineering Applications of Artificial Intelligence 133 \(2024\)\.
- \[31\]B\. B\. Mandelbrot, How long is the coast of britain?, Science 156 \(3775\) \(1967\)\.
- \[32\]T\. W\. Gamelin, R\. E\. Greene, Introduction to Topology: Second Edition, Courier Dover Publications, 2013\.
- \[33\]M\. E\. Houle, H\. Kashima, M\. Nett, Generalized expansion dimension, in: ICDMW, IEEE, 2012\.
- \[34\]D\. R\. Karger, M\. Ruhl, Finding nearest neighbors in growth\-restricted metrics, in: STOC, ACM, 2002\.
- \[35\]M\. E\. Houle, M\. Nett, Rank cover trees for nearest neighbor search, in: SISAP, Springer, 2013\.
- \[36\]M\. E\. Houle, X\. Ma, M\. Nett, V\. Oria, Dimensional testing for multi\-step similarity search, in: ICDM, IEEE, 2012\.
- \[37\]G\. Casanova, E\. Englmeier, M\. E\. Houle, P\. Kröger, M\. Nett, E\. Schubert, A\. Zimek, Dimensional testing for reverse k\-nearest neighbor search, Proc\. VLDB Endow\. 10 \(7\) \(2017\)\.
- \[38\]T\. de Vries, S\. Chawla, M\. E\. Houle, Density\-preserving projections for large\-scale local anomaly detection, Knowledge and Information Systems 32 \(1\) \(2012\)\.
- \[39\]A\. Anderberg, J\. Bailey, R\. J\. G\. B\. Campello, M\. E\. Houle, H\. O\. Marques, M\. Radovanovic, A\. Zimek, Dimensionality\-aware outlier detection, in: SIAM International Conference on Data Mining, 2024\.
- \[40\]X\. He, D\. Cai, P\. Niyogi, Laplacian score for feature selection, Advances in Neural Information Processing Systems 18, section 2 explicitly describes ”Variance” as the simplest unsupervised criterion: select features with largest variance \(2005\)\.
- \[41\]M\. A\. Hall, Correlation\-based feature selection for machine learningPhD thesis, University of Waikato\. Originally supervised, but the correlation part is frequently adapted to unsupervised scenarios \(1999\)\.
- \[42\]P\. Mitra, C\. A\. Murty, S\. K\. Pal, Unsupervised feature selection using feature similarity, IEEE Transactions on Pattern Analysis and Machine Intelligence 24 \(3\) \(2002\)\.[doi:10\.1109/34\.990132](https://doi.org/10.1109/34.990132)\.
- \[43\]H\. W\. Kuhn, The Hungarian Method for the assignment problem, Naval Research Logistics Quarterly 2 \(1\-2\) \(1955\)\.
- \[44\]J\. Li, K\. Cheng, S\. Wang, F\. Morstatter, R\. P\. Trevino, J\. Tang, H\. Liu, Feature selection: A data perspective \(2017\)\.
- \[45\]M\. Rajabinasab, A\. Zimek, FSEVAL: feature selection evaluation toolbox and dashboard, CoRR abs/2604\.18227 \(2026\)\.[arXiv:2604\.18227](http://arxiv.org/abs/2604.18227),[doi:10\.48550/ARXIV\.2604\.18227](https://doi.org/10.48550/ARXIV.2604.18227)\.
- \[46\]T\. M\. Cover, P\. E\. Hart, Nearest neighbor pattern classification, IEEE Transactions on Information Theory 13 \(1\) \(1967\)\.
- \[47\]L\. Breiman, J\. H\. Friedman, R\. A\. Olshen, C\. J\. Stone, Classification and Regression Trees, Wadsworth, 1984\.
- \[48\]C\. Cortes, V\. Vapnik, Support\-vector networks, Machine learning 20 \(3\) \(1995\)\.
- \[49\]L\. Breiman, Random Forests, Machine Learning 45 \(1\) \(2001\)\.
- \[50\]T\. Chen, C\. Guestrin, XGBoost: A scalable tree boosting system, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, 2016\.
- \[51\]M\. Rajabinasab, M\. E\. Houle, O\. Chelly, A\. Zimek, Worse than random: The importance of a baseline for unsupervised feature selection – extended version, CoRR abs/2605\.22973 \(2026\)\.[arXiv:2605\.22973](http://arxiv.org/abs/2605.22973),[doi:10\.48550/ARXIV\.2605\.22973](https://doi.org/10.48550/ARXIV.2605.22973)\.
- \[52\]J\. Demšar, Statistical comparisons of classifiers over multiple data sets, Journal of Machine Learning Research 7 \(2006\)\.
- \[53\]W\. Dong, C\. Moses, K\. Li, Efficient k\-nearest neighbor graph construction for generic similarity measures, in: WWW, ACM, 2011\.

## Appendix ASupplementary results

This appendix presents the results that are not included in the main text\. It consists of two parts\. Section[A\.1](https://arxiv.org/html/2610.40736#A1.SS1)presents the critical difference diagrams of the supervised metrics \(ACC and AUC\) for the KNN, decision tree, SVM, and XGBoost classifiers, and Section[A\.2](https://arxiv.org/html/2610.40736#A1.SS2)presents the performance plots on all 23 benchmark datasets\. All supervised results are reported for each of the five downstream classifiers separately, whereas the unsupervised metrics, namely clustering accuracy \(CLSACC\) and NMI, do not depend on the classifier and are therefore reported once\. The results for the random forest classifier and for the unsupervised metrics are presented in the main text, and they are not repeated here\.

### A\.1Critical difference diagrams

The critical difference diagrams are computed in the same way as in the main text, i\.e\., over all 23 benchmark datasets based on the average performance measured by the FSDEM score, using Wilcoxon\-Holm with a significance level of0\.20\.2\. The diagrams of the random forest classifier and of the unsupervised metrics for the first 10% and for the entire range of the number of selected features are presented in Figs\.[8](https://arxiv.org/html/2610.40736#S6.F8)and[6](https://arxiv.org/html/2610.40736#S6.F6)of the main text, respectively\. Each figure below contains the diagrams of ACC \(left\) and AUC \(right\) for one classifier and one range, and Table[A1](https://arxiv.org/html/2610.40736#A1.T1)provides an index of all figures, where each entry links to the corresponding figure\.

Table A1:Index of the supplementary critical difference diagrams\. Each entry gives the number of the figure with the diagrams of ACC and AUC for the corresponding classifier and range of the number of selected features \(up to 10% or 100% of the features\), and links to the figure\. Entries marked with†refer to the main text\.Range of the number of selected featuresClassifier / metrics0\.5% to 10%5% to 100%KNN \(ACC, AUC\)[A1](https://arxiv.org/html/2610.40736#A1.F1)[A2](https://arxiv.org/html/2610.40736#A1.F2)Decision tree \(ACC, AUC\)[A3](https://arxiv.org/html/2610.40736#A1.F3)[A4](https://arxiv.org/html/2610.40736#A1.F4)SVM \(ACC, AUC\)[A5](https://arxiv.org/html/2610.40736#A1.F5)[A6](https://arxiv.org/html/2610.40736#A1.F6)Random forest \(ACC, AUC\)[8](https://arxiv.org/html/2610.40736#S6.F8)†[6](https://arxiv.org/html/2610.40736#S6.F6)†XGBoost \(ACC, AUC\)[A7](https://arxiv.org/html/2610.40736#A1.F7)[A8](https://arxiv.org/html/2610.40736#A1.F8)CLSACC, NMI \(classifier independent\)[8](https://arxiv.org/html/2610.40736#S6.F8)†[6](https://arxiv.org/html/2610.40736#S6.F6)†Figure A1:Critical difference diagrams of ACC \(left\) and AUC \(right\) for the KNN classifier \(0\.5% to 10% of the features\)\.Figure A2:Critical difference diagrams of ACC \(left\) and AUC \(right\) for the KNN classifier \(5% to 100% of the features\)\.Figure A3:Critical difference diagrams of ACC \(left\) and AUC \(right\) for the decision tree classifier \(0\.5% to 10% of the features\)\.Figure A4:Critical difference diagrams of ACC \(left\) and AUC \(right\) for the decision tree classifier \(5% to 100% of the features\)\.Figure A5:Critical difference diagrams of ACC \(left\) and AUC \(right\) for the SVM classifier \(0\.5% to 10% of the features\)\.Figure A6:Critical difference diagrams of ACC \(left\) and AUC \(right\) for the SVM classifier \(5% to 100% of the features\)\.Figure A7:Critical difference diagrams of ACC \(left\) and AUC \(right\) for the XGBoost classifier \(0\.5% to 10% of the features\)\.Figure A8:Critical difference diagrams of ACC \(left\) and AUC \(right\) for the XGBoost classifier \(5% to 100% of the features\)\.
### A\.2Performance plots

This part presents the performance plots on all 23 benchmark datasets and for both experimental settings, in which the number of selected features ranges from 0\.5% to 10% and from 5% to 100% of the features, respectively\. The unsupervised metrics \(CLSACC and NMI\) are reported once for each dataset, whereas the supervised metrics \(ACC and AUC\) are reported separately for each of the five classifiers, i\.e\., KNN, decision tree, SVM, random forest, and XGBoost\. In all plots, the random baseline is the average of 100 independent runs, and the shaded area denotes one standard deviation\. The results on the Isolet dataset for the random forest classifier and for the unsupervised metrics are presented in Figs\.[4](https://arxiv.org/html/2610.40736#S6.F4)and[3](https://arxiv.org/html/2610.40736#S6.F3)of the main text for the first 10% and for the entire range of the number of selected features, respectively, and are not repeated here\. Table[A2](https://arxiv.org/html/2610.40736#A1.T2)provides an index of all plots, where each entry links to the corresponding figure\.

Table A2:Index of the supplementary performance plots\. Each entry gives the number of the figure for the corresponding dataset, metric group or classifier, and range of the number of selected features \(up to 10% or 100% of the features\), and links to the figure\. Entries marked with†refer to the main text\.UnsupervisedSupervised \(ACC, AUC\)CLSACC, NMIKNNDecision treeSVMRandom forestXGBoostDataset10%100%10%100%10%100%10%100%10%100%10%100%ALLAML[A9](https://arxiv.org/html/2610.40736#A1.F9)[A31](https://arxiv.org/html/2610.40736#A1.F31)[A53](https://arxiv.org/html/2610.40736#A1.F53)[A76](https://arxiv.org/html/2610.40736#A1.F76)[A99](https://arxiv.org/html/2610.40736#A1.F99)[A122](https://arxiv.org/html/2610.40736#A1.F122)[A145](https://arxiv.org/html/2610.40736#A1.F145)[A168](https://arxiv.org/html/2610.40736#A1.F168)[A191](https://arxiv.org/html/2610.40736#A1.F191)[A213](https://arxiv.org/html/2610.40736#A1.F213)[A235](https://arxiv.org/html/2610.40736#A1.F235)[A258](https://arxiv.org/html/2610.40736#A1.F258)arcene[A10](https://arxiv.org/html/2610.40736#A1.F10)[A32](https://arxiv.org/html/2610.40736#A1.F32)[A54](https://arxiv.org/html/2610.40736#A1.F54)[A77](https://arxiv.org/html/2610.40736#A1.F77)[A100](https://arxiv.org/html/2610.40736#A1.F100)[A123](https://arxiv.org/html/2610.40736#A1.F123)[A146](https://arxiv.org/html/2610.40736#A1.F146)[A169](https://arxiv.org/html/2610.40736#A1.F169)[A192](https://arxiv.org/html/2610.40736#A1.F192)[A214](https://arxiv.org/html/2610.40736#A1.F214)[A236](https://arxiv.org/html/2610.40736#A1.F236)[A259](https://arxiv.org/html/2610.40736#A1.F259)Carcinom[A11](https://arxiv.org/html/2610.40736#A1.F11)[A33](https://arxiv.org/html/2610.40736#A1.F33)[A55](https://arxiv.org/html/2610.40736#A1.F55)[A78](https://arxiv.org/html/2610.40736#A1.F78)[A101](https://arxiv.org/html/2610.40736#A1.F101)[A124](https://arxiv.org/html/2610.40736#A1.F124)[A147](https://arxiv.org/html/2610.40736#A1.F147)[A170](https://arxiv.org/html/2610.40736#A1.F170)[A193](https://arxiv.org/html/2610.40736#A1.F193)[A215](https://arxiv.org/html/2610.40736#A1.F215)[A237](https://arxiv.org/html/2610.40736#A1.F237)[A260](https://arxiv.org/html/2610.40736#A1.F260)CLL\_SUB\_111[A12](https://arxiv.org/html/2610.40736#A1.F12)[A34](https://arxiv.org/html/2610.40736#A1.F34)[A56](https://arxiv.org/html/2610.40736#A1.F56)[A79](https://arxiv.org/html/2610.40736#A1.F79)[A102](https://arxiv.org/html/2610.40736#A1.F102)[A125](https://arxiv.org/html/2610.40736#A1.F125)[A148](https://arxiv.org/html/2610.40736#A1.F148)[A171](https://arxiv.org/html/2610.40736#A1.F171)[A194](https://arxiv.org/html/2610.40736#A1.F194)[A216](https://arxiv.org/html/2610.40736#A1.F216)[A238](https://arxiv.org/html/2610.40736#A1.F238)[A261](https://arxiv.org/html/2610.40736#A1.F261)COIL20[A13](https://arxiv.org/html/2610.40736#A1.F13)[A35](https://arxiv.org/html/2610.40736#A1.F35)[A57](https://arxiv.org/html/2610.40736#A1.F57)[A80](https://arxiv.org/html/2610.40736#A1.F80)[A103](https://arxiv.org/html/2610.40736#A1.F103)[A126](https://arxiv.org/html/2610.40736#A1.F126)[A149](https://arxiv.org/html/2610.40736#A1.F149)[A172](https://arxiv.org/html/2610.40736#A1.F172)[A195](https://arxiv.org/html/2610.40736#A1.F195)[A217](https://arxiv.org/html/2610.40736#A1.F217)[A239](https://arxiv.org/html/2610.40736#A1.F239)[A262](https://arxiv.org/html/2610.40736#A1.F262)colon[A14](https://arxiv.org/html/2610.40736#A1.F14)[A36](https://arxiv.org/html/2610.40736#A1.F36)[A58](https://arxiv.org/html/2610.40736#A1.F58)[A81](https://arxiv.org/html/2610.40736#A1.F81)[A104](https://arxiv.org/html/2610.40736#A1.F104)[A127](https://arxiv.org/html/2610.40736#A1.F127)[A150](https://arxiv.org/html/2610.40736#A1.F150)[A173](https://arxiv.org/html/2610.40736#A1.F173)[A196](https://arxiv.org/html/2610.40736#A1.F196)[A218](https://arxiv.org/html/2610.40736#A1.F218)[A240](https://arxiv.org/html/2610.40736#A1.F240)[A263](https://arxiv.org/html/2610.40736#A1.F263)gisette[A15](https://arxiv.org/html/2610.40736#A1.F15)[A37](https://arxiv.org/html/2610.40736#A1.F37)[A59](https://arxiv.org/html/2610.40736#A1.F59)[A82](https://arxiv.org/html/2610.40736#A1.F82)[A105](https://arxiv.org/html/2610.40736#A1.F105)[A128](https://arxiv.org/html/2610.40736#A1.F128)[A151](https://arxiv.org/html/2610.40736#A1.F151)[A174](https://arxiv.org/html/2610.40736#A1.F174)[A197](https://arxiv.org/html/2610.40736#A1.F197)[A219](https://arxiv.org/html/2610.40736#A1.F219)[A241](https://arxiv.org/html/2610.40736#A1.F241)[A264](https://arxiv.org/html/2610.40736#A1.F264)GLI\_85[A16](https://arxiv.org/html/2610.40736#A1.F16)[A38](https://arxiv.org/html/2610.40736#A1.F38)[A60](https://arxiv.org/html/2610.40736#A1.F60)[A83](https://arxiv.org/html/2610.40736#A1.F83)[A106](https://arxiv.org/html/2610.40736#A1.F106)[A129](https://arxiv.org/html/2610.40736#A1.F129)[A152](https://arxiv.org/html/2610.40736#A1.F152)[A175](https://arxiv.org/html/2610.40736#A1.F175)[A198](https://arxiv.org/html/2610.40736#A1.F198)[A220](https://arxiv.org/html/2610.40736#A1.F220)[A242](https://arxiv.org/html/2610.40736#A1.F242)[A265](https://arxiv.org/html/2610.40736#A1.F265)GLIOMA[A17](https://arxiv.org/html/2610.40736#A1.F17)[A39](https://arxiv.org/html/2610.40736#A1.F39)[A61](https://arxiv.org/html/2610.40736#A1.F61)[A84](https://arxiv.org/html/2610.40736#A1.F84)[A107](https://arxiv.org/html/2610.40736#A1.F107)[A130](https://arxiv.org/html/2610.40736#A1.F130)[A153](https://arxiv.org/html/2610.40736#A1.F153)[A176](https://arxiv.org/html/2610.40736#A1.F176)[A199](https://arxiv.org/html/2610.40736#A1.F199)[A221](https://arxiv.org/html/2610.40736#A1.F221)[A243](https://arxiv.org/html/2610.40736#A1.F243)[A266](https://arxiv.org/html/2610.40736#A1.F266)Isolet[4](https://arxiv.org/html/2610.40736#S6.F4)†[3](https://arxiv.org/html/2610.40736#S6.F3)†[A62](https://arxiv.org/html/2610.40736#A1.F62)[A85](https://arxiv.org/html/2610.40736#A1.F85)[A108](https://arxiv.org/html/2610.40736#A1.F108)[A131](https://arxiv.org/html/2610.40736#A1.F131)[A154](https://arxiv.org/html/2610.40736#A1.F154)[A177](https://arxiv.org/html/2610.40736#A1.F177)[4](https://arxiv.org/html/2610.40736#S6.F4)†[3](https://arxiv.org/html/2610.40736#S6.F3)†[A244](https://arxiv.org/html/2610.40736#A1.F244)[A267](https://arxiv.org/html/2610.40736#A1.F267)leukemia[A18](https://arxiv.org/html/2610.40736#A1.F18)[A40](https://arxiv.org/html/2610.40736#A1.F40)[A63](https://arxiv.org/html/2610.40736#A1.F63)[A86](https://arxiv.org/html/2610.40736#A1.F86)[A109](https://arxiv.org/html/2610.40736#A1.F109)[A132](https://arxiv.org/html/2610.40736#A1.F132)[A155](https://arxiv.org/html/2610.40736#A1.F155)[A178](https://arxiv.org/html/2610.40736#A1.F178)[A200](https://arxiv.org/html/2610.40736#A1.F200)[A222](https://arxiv.org/html/2610.40736#A1.F222)[A245](https://arxiv.org/html/2610.40736#A1.F245)[A268](https://arxiv.org/html/2610.40736#A1.F268)lung[A19](https://arxiv.org/html/2610.40736#A1.F19)[A41](https://arxiv.org/html/2610.40736#A1.F41)[A64](https://arxiv.org/html/2610.40736#A1.F64)[A87](https://arxiv.org/html/2610.40736#A1.F87)[A110](https://arxiv.org/html/2610.40736#A1.F110)[A133](https://arxiv.org/html/2610.40736#A1.F133)[A156](https://arxiv.org/html/2610.40736#A1.F156)[A179](https://arxiv.org/html/2610.40736#A1.F179)[A201](https://arxiv.org/html/2610.40736#A1.F201)[A223](https://arxiv.org/html/2610.40736#A1.F223)[A246](https://arxiv.org/html/2610.40736#A1.F246)[A269](https://arxiv.org/html/2610.40736#A1.F269)lung\_discrete[A20](https://arxiv.org/html/2610.40736#A1.F20)[A42](https://arxiv.org/html/2610.40736#A1.F42)[A65](https://arxiv.org/html/2610.40736#A1.F65)[A88](https://arxiv.org/html/2610.40736#A1.F88)[A111](https://arxiv.org/html/2610.40736#A1.F111)[A134](https://arxiv.org/html/2610.40736#A1.F134)[A157](https://arxiv.org/html/2610.40736#A1.F157)[A180](https://arxiv.org/html/2610.40736#A1.F180)[A202](https://arxiv.org/html/2610.40736#A1.F202)[A224](https://arxiv.org/html/2610.40736#A1.F224)[A247](https://arxiv.org/html/2610.40736#A1.F247)[A270](https://arxiv.org/html/2610.40736#A1.F270)madelon[A21](https://arxiv.org/html/2610.40736#A1.F21)[A43](https://arxiv.org/html/2610.40736#A1.F43)[A66](https://arxiv.org/html/2610.40736#A1.F66)[A89](https://arxiv.org/html/2610.40736#A1.F89)[A112](https://arxiv.org/html/2610.40736#A1.F112)[A135](https://arxiv.org/html/2610.40736#A1.F135)[A158](https://arxiv.org/html/2610.40736#A1.F158)[A181](https://arxiv.org/html/2610.40736#A1.F181)[A203](https://arxiv.org/html/2610.40736#A1.F203)[A225](https://arxiv.org/html/2610.40736#A1.F225)[A248](https://arxiv.org/html/2610.40736#A1.F248)[A271](https://arxiv.org/html/2610.40736#A1.F271)ORL[A22](https://arxiv.org/html/2610.40736#A1.F22)[A44](https://arxiv.org/html/2610.40736#A1.F44)[A67](https://arxiv.org/html/2610.40736#A1.F67)[A90](https://arxiv.org/html/2610.40736#A1.F90)[A113](https://arxiv.org/html/2610.40736#A1.F113)[A136](https://arxiv.org/html/2610.40736#A1.F136)[A159](https://arxiv.org/html/2610.40736#A1.F159)[A182](https://arxiv.org/html/2610.40736#A1.F182)[A204](https://arxiv.org/html/2610.40736#A1.F204)[A226](https://arxiv.org/html/2610.40736#A1.F226)[A249](https://arxiv.org/html/2610.40736#A1.F249)[A272](https://arxiv.org/html/2610.40736#A1.F272)orlraws10P[A23](https://arxiv.org/html/2610.40736#A1.F23)[A45](https://arxiv.org/html/2610.40736#A1.F45)[A68](https://arxiv.org/html/2610.40736#A1.F68)[A91](https://arxiv.org/html/2610.40736#A1.F91)[A114](https://arxiv.org/html/2610.40736#A1.F114)[A137](https://arxiv.org/html/2610.40736#A1.F137)[A160](https://arxiv.org/html/2610.40736#A1.F160)[A183](https://arxiv.org/html/2610.40736#A1.F183)[A205](https://arxiv.org/html/2610.40736#A1.F205)[A227](https://arxiv.org/html/2610.40736#A1.F227)[A250](https://arxiv.org/html/2610.40736#A1.F250)[A273](https://arxiv.org/html/2610.40736#A1.F273)pixraw10P[A24](https://arxiv.org/html/2610.40736#A1.F24)[A46](https://arxiv.org/html/2610.40736#A1.F46)[A69](https://arxiv.org/html/2610.40736#A1.F69)[A92](https://arxiv.org/html/2610.40736#A1.F92)[A115](https://arxiv.org/html/2610.40736#A1.F115)[A138](https://arxiv.org/html/2610.40736#A1.F138)[A161](https://arxiv.org/html/2610.40736#A1.F161)[A184](https://arxiv.org/html/2610.40736#A1.F184)[A206](https://arxiv.org/html/2610.40736#A1.F206)[A228](https://arxiv.org/html/2610.40736#A1.F228)[A251](https://arxiv.org/html/2610.40736#A1.F251)[A274](https://arxiv.org/html/2610.40736#A1.F274)Prostate\_GE[A25](https://arxiv.org/html/2610.40736#A1.F25)[A47](https://arxiv.org/html/2610.40736#A1.F47)[A70](https://arxiv.org/html/2610.40736#A1.F70)[A93](https://arxiv.org/html/2610.40736#A1.F93)[A116](https://arxiv.org/html/2610.40736#A1.F116)[A139](https://arxiv.org/html/2610.40736#A1.F139)[A162](https://arxiv.org/html/2610.40736#A1.F162)[A185](https://arxiv.org/html/2610.40736#A1.F185)[A207](https://arxiv.org/html/2610.40736#A1.F207)[A229](https://arxiv.org/html/2610.40736#A1.F229)[A252](https://arxiv.org/html/2610.40736#A1.F252)[A275](https://arxiv.org/html/2610.40736#A1.F275)SMK\_CAN\_187[A26](https://arxiv.org/html/2610.40736#A1.F26)[A48](https://arxiv.org/html/2610.40736#A1.F48)[A71](https://arxiv.org/html/2610.40736#A1.F71)[A94](https://arxiv.org/html/2610.40736#A1.F94)[A117](https://arxiv.org/html/2610.40736#A1.F117)[A140](https://arxiv.org/html/2610.40736#A1.F140)[A163](https://arxiv.org/html/2610.40736#A1.F163)[A186](https://arxiv.org/html/2610.40736#A1.F186)[A208](https://arxiv.org/html/2610.40736#A1.F208)[A230](https://arxiv.org/html/2610.40736#A1.F230)[A253](https://arxiv.org/html/2610.40736#A1.F253)[A276](https://arxiv.org/html/2610.40736#A1.F276)TOX\_171[A27](https://arxiv.org/html/2610.40736#A1.F27)[A49](https://arxiv.org/html/2610.40736#A1.F49)[A72](https://arxiv.org/html/2610.40736#A1.F72)[A95](https://arxiv.org/html/2610.40736#A1.F95)[A118](https://arxiv.org/html/2610.40736#A1.F118)[A141](https://arxiv.org/html/2610.40736#A1.F141)[A164](https://arxiv.org/html/2610.40736#A1.F164)[A187](https://arxiv.org/html/2610.40736#A1.F187)[A209](https://arxiv.org/html/2610.40736#A1.F209)[A231](https://arxiv.org/html/2610.40736#A1.F231)[A254](https://arxiv.org/html/2610.40736#A1.F254)[A277](https://arxiv.org/html/2610.40736#A1.F277)warpAR10P[A28](https://arxiv.org/html/2610.40736#A1.F28)[A50](https://arxiv.org/html/2610.40736#A1.F50)[A73](https://arxiv.org/html/2610.40736#A1.F73)[A96](https://arxiv.org/html/2610.40736#A1.F96)[A119](https://arxiv.org/html/2610.40736#A1.F119)[A142](https://arxiv.org/html/2610.40736#A1.F142)[A165](https://arxiv.org/html/2610.40736#A1.F165)[A188](https://arxiv.org/html/2610.40736#A1.F188)[A210](https://arxiv.org/html/2610.40736#A1.F210)[A232](https://arxiv.org/html/2610.40736#A1.F232)[A255](https://arxiv.org/html/2610.40736#A1.F255)[A278](https://arxiv.org/html/2610.40736#A1.F278)warpPIE10P[A29](https://arxiv.org/html/2610.40736#A1.F29)[A51](https://arxiv.org/html/2610.40736#A1.F51)[A74](https://arxiv.org/html/2610.40736#A1.F74)[A97](https://arxiv.org/html/2610.40736#A1.F97)[A120](https://arxiv.org/html/2610.40736#A1.F120)[A143](https://arxiv.org/html/2610.40736#A1.F143)[A166](https://arxiv.org/html/2610.40736#A1.F166)[A189](https://arxiv.org/html/2610.40736#A1.F189)[A211](https://arxiv.org/html/2610.40736#A1.F211)[A233](https://arxiv.org/html/2610.40736#A1.F233)[A256](https://arxiv.org/html/2610.40736#A1.F256)[A279](https://arxiv.org/html/2610.40736#A1.F279)Yale[A30](https://arxiv.org/html/2610.40736#A1.F30)[A52](https://arxiv.org/html/2610.40736#A1.F52)[A75](https://arxiv.org/html/2610.40736#A1.F75)[A98](https://arxiv.org/html/2610.40736#A1.F98)[A121](https://arxiv.org/html/2610.40736#A1.F121)[A144](https://arxiv.org/html/2610.40736#A1.F144)[A167](https://arxiv.org/html/2610.40736#A1.F167)[A190](https://arxiv.org/html/2610.40736#A1.F190)[A212](https://arxiv.org/html/2610.40736#A1.F212)[A234](https://arxiv.org/html/2610.40736#A1.F234)[A257](https://arxiv.org/html/2610.40736#A1.F257)[A280](https://arxiv.org/html/2610.40736#A1.F280)Figure A9:CLSACC and NMI on the ALLAML dataset \(0\.5% to 10% of the features\)\.Figure A10:CLSACC and NMI on the arcene dataset \(0\.5% to 10% of the features\)\.Figure A11:CLSACC and NMI on the Carcinom dataset \(0\.5% to 10% of the features\)\.Figure A12:CLSACC and NMI on the CLL\_SUB\_111 dataset \(0\.5% to 10% of the features\)\.Figure A13:CLSACC and NMI on the COIL20 dataset \(0\.5% to 10% of the features\)\.Figure A14:CLSACC and NMI on the colon dataset \(0\.5% to 10% of the features\)\.Figure A15:CLSACC and NMI on the gisette dataset \(0\.5% to 10% of the features\)\.Figure A16:CLSACC and NMI on the GLI\_85 dataset \(0\.5% to 10% of the features\)\.Figure A17:CLSACC and NMI on the GLIOMA dataset \(0\.5% to 10% of the features\)\.Figure A18:CLSACC and NMI on the leukemia dataset \(0\.5% to 10% of the features\)\.Figure A19:CLSACC and NMI on the lung dataset \(0\.5% to 10% of the features\)\.Figure A20:CLSACC and NMI on the lung\_discrete dataset \(0\.5% to 10% of the features\)\.Figure A21:CLSACC and NMI on the madelon dataset \(0\.5% to 10% of the features\)\.Figure A22:CLSACC and NMI on the ORL dataset \(0\.5% to 10% of the features\)\.Figure A23:CLSACC and NMI on the orlraws10P dataset \(0\.5% to 10% of the features\)\.Figure A24:CLSACC and NMI on the pixraw10P dataset \(0\.5% to 10% of the features\)\.Figure A25:CLSACC and NMI on the Prostate\_GE dataset \(0\.5% to 10% of the features\)\.Figure A26:CLSACC and NMI on the SMK\_CAN\_187 dataset \(0\.5% to 10% of the features\)\.Figure A27:CLSACC and NMI on the TOX\_171 dataset \(0\.5% to 10% of the features\)\.Figure A28:CLSACC and NMI on the warpAR10P dataset \(0\.5% to 10% of the features\)\.Figure A29:CLSACC and NMI on the warpPIE10P dataset \(0\.5% to 10% of the features\)\.Figure A30:CLSACC and NMI on the Yale dataset \(0\.5% to 10% of the features\)\.Figure A31:CLSACC and NMI on the ALLAML dataset \(5% to 100% of the features\)\.Figure A32:CLSACC and NMI on the arcene dataset \(5% to 100% of the features\)\.Figure A33:CLSACC and NMI on the Carcinom dataset \(5% to 100% of the features\)\.Figure A34:CLSACC and NMI on the CLL\_SUB\_111 dataset \(5% to 100% of the features\)\.Figure A35:CLSACC and NMI on the COIL20 dataset \(5% to 100% of the features\)\.Figure A36:CLSACC and NMI on the colon dataset \(5% to 100% of the features\)\.Figure A37:CLSACC and NMI on the gisette dataset \(5% to 100% of the features\)\.Figure A38:CLSACC and NMI on the GLI\_85 dataset \(5% to 100% of the features\)\.Figure A39:CLSACC and NMI on the GLIOMA dataset \(5% to 100% of the features\)\.Figure A40:CLSACC and NMI on the leukemia dataset \(5% to 100% of the features\)\.Figure A41:CLSACC and NMI on the lung dataset \(5% to 100% of the features\)\.Figure A42:CLSACC and NMI on the lung\_discrete dataset \(5% to 100% of the features\)\.Figure A43:CLSACC and NMI on the madelon dataset \(5% to 100% of the features\)\.Figure A44:CLSACC and NMI on the ORL dataset \(5% to 100% of the features\)\.Figure A45:CLSACC and NMI on the orlraws10P dataset \(5% to 100% of the features\)\.Figure A46:CLSACC and NMI on the pixraw10P dataset \(5% to 100% of the features\)\.Figure A47:CLSACC and NMI on the Prostate\_GE dataset \(5% to 100% of the features\)\.Figure A48:CLSACC and NMI on the SMK\_CAN\_187 dataset \(5% to 100% of the features\)\.Figure A49:CLSACC and NMI on the TOX\_171 dataset \(5% to 100% of the features\)\.Figure A50:CLSACC and NMI on the warpAR10P dataset \(5% to 100% of the features\)\.Figure A51:CLSACC and NMI on the warpPIE10P dataset \(5% to 100% of the features\)\.Figure A52:CLSACC and NMI on the Yale dataset \(5% to 100% of the features\)\.Figure A53:ACC and AUC of the KNN classifier on the ALLAML dataset \(0\.5% to 10% of the features\)\.Figure A54:ACC and AUC of the KNN classifier on the arcene dataset \(0\.5% to 10% of the features\)\.Figure A55:ACC and AUC of the KNN classifier on the Carcinom dataset \(0\.5% to 10% of the features\)\.Figure A56:ACC and AUC of the KNN classifier on the CLL\_SUB\_111 dataset \(0\.5% to 10% of the features\)\.Figure A57:ACC and AUC of the KNN classifier on the COIL20 dataset \(0\.5% to 10% of the features\)\.Figure A58:ACC and AUC of the KNN classifier on the colon dataset \(0\.5% to 10% of the features\)\.Figure A59:ACC and AUC of the KNN classifier on the gisette dataset \(0\.5% to 10% of the features\)\.Figure A60:ACC and AUC of the KNN classifier on the GLI\_85 dataset \(0\.5% to 10% of the features\)\.Figure A61:ACC and AUC of the KNN classifier on the GLIOMA dataset \(0\.5% to 10% of the features\)\.Figure A62:ACC and AUC of the KNN classifier on the Isolet dataset \(0\.5% to 10% of the features\)\.Figure A63:ACC and AUC of the KNN classifier on the leukemia dataset \(0\.5% to 10% of the features\)\.Figure A64:ACC and AUC of the KNN classifier on the lung dataset \(0\.5% to 10% of the features\)\.Figure A65:ACC and AUC of the KNN classifier on the lung\_discrete dataset \(0\.5% to 10% of the features\)\.Figure A66:ACC and AUC of the KNN classifier on the madelon dataset \(0\.5% to 10% of the features\)\.Figure A67:ACC and AUC of the KNN classifier on the ORL dataset \(0\.5% to 10% of the features\)\.Figure A68:ACC and AUC of the KNN classifier on the orlraws10P dataset \(0\.5% to 10% of the features\)\.Figure A69:ACC and AUC of the KNN classifier on the pixraw10P dataset \(0\.5% to 10% of the features\)\.Figure A70:ACC and AUC of the KNN classifier on the Prostate\_GE dataset \(0\.5% to 10% of the features\)\.Figure A71:ACC and AUC of the KNN classifier on the SMK\_CAN\_187 dataset \(0\.5% to 10% of the features\)\.Figure A72:ACC and AUC of the KNN classifier on the TOX\_171 dataset \(0\.5% to 10% of the features\)\.Figure A73:ACC and AUC of the KNN classifier on the warpAR10P dataset \(0\.5% to 10% of the features\)\.Figure A74:ACC and AUC of the KNN classifier on the warpPIE10P dataset \(0\.5% to 10% of the features\)\.Figure A75:ACC and AUC of the KNN classifier on the Yale dataset \(0\.5% to 10% of the features\)\.Figure A76:ACC and AUC of the KNN classifier on the ALLAML dataset \(5% to 100% of the features\)\.Figure A77:ACC and AUC of the KNN classifier on the arcene dataset \(5% to 100% of the features\)\.Figure A78:ACC and AUC of the KNN classifier on the Carcinom dataset \(5% to 100% of the features\)\.Figure A79:ACC and AUC of the KNN classifier on the CLL\_SUB\_111 dataset \(5% to 100% of the features\)\.Figure A80:ACC and AUC of the KNN classifier on the COIL20 dataset \(5% to 100% of the features\)\.Figure A81:ACC and AUC of the KNN classifier on the colon dataset \(5% to 100% of the features\)\.Figure A82:ACC and AUC of the KNN classifier on the gisette dataset \(5% to 100% of the features\)\.Figure A83:ACC and AUC of the KNN classifier on the GLI\_85 dataset \(5% to 100% of the features\)\.Figure A84:ACC and AUC of the KNN classifier on the GLIOMA dataset \(5% to 100% of the features\)\.Figure A85:ACC and AUC of the KNN classifier on the Isolet dataset \(5% to 100% of the features\)\.Figure A86:ACC and AUC of the KNN classifier on the leukemia dataset \(5% to 100% of the features\)\.Figure A87:ACC and AUC of the KNN classifier on the lung dataset \(5% to 100% of the features\)\.Figure A88:ACC and AUC of the KNN classifier on the lung\_discrete dataset \(5% to 100% of the features\)\.Figure A89:ACC and AUC of the KNN classifier on the madelon dataset \(5% to 100% of the features\)\.Figure A90:ACC and AUC of the KNN classifier on the ORL dataset \(5% to 100% of the features\)\.Figure A91:ACC and AUC of the KNN classifier on the orlraws10P dataset \(5% to 100% of the features\)\.Figure A92:ACC and AUC of the KNN classifier on the pixraw10P dataset \(5% to 100% of the features\)\.Figure A93:ACC and AUC of the KNN classifier on the Prostate\_GE dataset \(5% to 100% of the features\)\.Figure A94:ACC and AUC of the KNN classifier on the SMK\_CAN\_187 dataset \(5% to 100% of the features\)\.Figure A95:ACC and AUC of the KNN classifier on the TOX\_171 dataset \(5% to 100% of the features\)\.Figure A96:ACC and AUC of the KNN classifier on the warpAR10P dataset \(5% to 100% of the features\)\.Figure A97:ACC and AUC of the KNN classifier on the warpPIE10P dataset \(5% to 100% of the features\)\.Figure A98:ACC and AUC of the KNN classifier on the Yale dataset \(5% to 100% of the features\)\.Figure A99:ACC and AUC of the decision tree classifier on the ALLAML dataset \(0\.5% to 10% of the features\)\.Figure A100:ACC and AUC of the decision tree classifier on the arcene dataset \(0\.5% to 10% of the features\)\.Figure A101:ACC and AUC of the decision tree classifier on the Carcinom dataset \(0\.5% to 10% of the features\)\.Figure A102:ACC and AUC of the decision tree classifier on the CLL\_SUB\_111 dataset \(0\.5% to 10% of the features\)\.Figure A103:ACC and AUC of the decision tree classifier on the COIL20 dataset \(0\.5% to 10% of the features\)\.Figure A104:ACC and AUC of the decision tree classifier on the colon dataset \(0\.5% to 10% of the features\)\.Figure A105:ACC and AUC of the decision tree classifier on the gisette dataset \(0\.5% to 10% of the features\)\.Figure A106:ACC and AUC of the decision tree classifier on the GLI\_85 dataset \(0\.5% to 10% of the features\)\.Figure A107:ACC and AUC of the decision tree classifier on the GLIOMA dataset \(0\.5% to 10% of the features\)\.Figure A108:ACC and AUC of the decision tree classifier on the Isolet dataset \(0\.5% to 10% of the features\)\.Figure A109:ACC and AUC of the decision tree classifier on the leukemia dataset \(0\.5% to 10% of the features\)\.Figure A110:ACC and AUC of the decision tree classifier on the lung dataset \(0\.5% to 10% of the features\)\.Figure A111:ACC and AUC of the decision tree classifier on the lung\_discrete dataset \(0\.5% to 10% of the features\)\.Figure A112:ACC and AUC of the decision tree classifier on the madelon dataset \(0\.5% to 10% of the features\)\.Figure A113:ACC and AUC of the decision tree classifier on the ORL dataset \(0\.5% to 10% of the features\)\.Figure A114:ACC and AUC of the decision tree classifier on the orlraws10P dataset \(0\.5% to 10% of the features\)\.Figure A115:ACC and AUC of the decision tree classifier on the pixraw10P dataset \(0\.5% to 10% of the features\)\.Figure A116:ACC and AUC of the decision tree classifier on the Prostate\_GE dataset \(0\.5% to 10% of the features\)\.Figure A117:ACC and AUC of the decision tree classifier on the SMK\_CAN\_187 dataset \(0\.5% to 10% of the features\)\.Figure A118:ACC and AUC of the decision tree classifier on the TOX\_171 dataset \(0\.5% to 10% of the features\)\.Figure A119:ACC and AUC of the decision tree classifier on the warpAR10P dataset \(0\.5% to 10% of the features\)\.Figure A120:ACC and AUC of the decision tree classifier on the warpPIE10P dataset \(0\.5% to 10% of the features\)\.Figure A121:ACC and AUC of the decision tree classifier on the Yale dataset \(0\.5% to 10% of the features\)\.Figure A122:ACC and AUC of the decision tree classifier on the ALLAML dataset \(5% to 100% of the features\)\.Figure A123:ACC and AUC of the decision tree classifier on the arcene dataset \(5% to 100% of the features\)\.Figure A124:ACC and AUC of the decision tree classifier on the Carcinom dataset \(5% to 100% of the features\)\.Figure A125:ACC and AUC of the decision tree classifier on the CLL\_SUB\_111 dataset \(5% to 100% of the features\)\.Figure A126:ACC and AUC of the decision tree classifier on the COIL20 dataset \(5% to 100% of the features\)\.Figure A127:ACC and AUC of the decision tree classifier on the colon dataset \(5% to 100% of the features\)\.Figure A128:ACC and AUC of the decision tree classifier on the gisette dataset \(5% to 100% of the features\)\.Figure A129:ACC and AUC of the decision tree classifier on the GLI\_85 dataset \(5% to 100% of the features\)\.Figure A130:ACC and AUC of the decision tree classifier on the GLIOMA dataset \(5% to 100% of the features\)\.Figure A131:ACC and AUC of the decision tree classifier on the Isolet dataset \(5% to 100% of the features\)\.Figure A132:ACC and AUC of the decision tree classifier on the leukemia dataset \(5% to 100% of the features\)\.Figure A133:ACC and AUC of the decision tree classifier on the lung dataset \(5% to 100% of the features\)\.Figure A134:ACC and AUC of the decision tree classifier on the lung\_discrete dataset \(5% to 100% of the features\)\.Figure A135:ACC and AUC of the decision tree classifier on the madelon dataset \(5% to 100% of the features\)\.Figure A136:ACC and AUC of the decision tree classifier on the ORL dataset \(5% to 100% of the features\)\.Figure A137:ACC and AUC of the decision tree classifier on the orlraws10P dataset \(5% to 100% of the features\)\.Figure A138:ACC and AUC of the decision tree classifier on the pixraw10P dataset \(5% to 100% of the features\)\.Figure A139:ACC and AUC of the decision tree classifier on the Prostate\_GE dataset \(5% to 100% of the features\)\.Figure A140:ACC and AUC of the decision tree classifier on the SMK\_CAN\_187 dataset \(5% to 100% of the features\)\.Figure A141:ACC and AUC of the decision tree classifier on the TOX\_171 dataset \(5% to 100% of the features\)\.Figure A142:ACC and AUC of the decision tree classifier on the warpAR10P dataset \(5% to 100% of the features\)\.Figure A143:ACC and AUC of the decision tree classifier on the warpPIE10P dataset \(5% to 100% of the features\)\.Figure A144:ACC and AUC of the decision tree classifier on the Yale dataset \(5% to 100% of the features\)\.Figure A145:ACC and AUC of the SVM classifier on the ALLAML dataset \(0\.5% to 10% of the features\)\.Figure A146:ACC and AUC of the SVM classifier on the arcene dataset \(0\.5% to 10% of the features\)\.Figure A147:ACC and AUC of the SVM classifier on the Carcinom dataset \(0\.5% to 10% of the features\)\.Figure A148:ACC and AUC of the SVM classifier on the CLL\_SUB\_111 dataset \(0\.5% to 10% of the features\)\.Figure A149:ACC and AUC of the SVM classifier on the COIL20 dataset \(0\.5% to 10% of the features\)\.Figure A150:ACC and AUC of the SVM classifier on the colon dataset \(0\.5% to 10% of the features\)\.Figure A151:ACC and AUC of the SVM classifier on the gisette dataset \(0\.5% to 10% of the features\)\.Figure A152:ACC and AUC of the SVM classifier on the GLI\_85 dataset \(0\.5% to 10% of the features\)\.Figure A153:ACC and AUC of the SVM classifier on the GLIOMA dataset \(0\.5% to 10% of the features\)\.Figure A154:ACC and AUC of the SVM classifier on the Isolet dataset \(0\.5% to 10% of the features\)\.Figure A155:ACC and AUC of the SVM classifier on the leukemia dataset \(0\.5% to 10% of the features\)\.Figure A156:ACC and AUC of the SVM classifier on the lung dataset \(0\.5% to 10% of the features\)\.Figure A157:ACC and AUC of the SVM classifier on the lung\_discrete dataset \(0\.5% to 10% of the features\)\.Figure A158:ACC and AUC of the SVM classifier on the madelon dataset \(0\.5% to 10% of the features\)\.Figure A159:ACC and AUC of the SVM classifier on the ORL dataset \(0\.5% to 10% of the features\)\.Figure A160:ACC and AUC of the SVM classifier on the orlraws10P dataset \(0\.5% to 10% of the features\)\.Figure A161:ACC and AUC of the SVM classifier on the pixraw10P dataset \(0\.5% to 10% of the features\)\.Figure A162:ACC and AUC of the SVM classifier on the Prostate\_GE dataset \(0\.5% to 10% of the features\)\.Figure A163:ACC and AUC of the SVM classifier on the SMK\_CAN\_187 dataset \(0\.5% to 10% of the features\)\.Figure A164:ACC and AUC of the SVM classifier on the TOX\_171 dataset \(0\.5% to 10% of the features\)\.Figure A165:ACC and AUC of the SVM classifier on the warpAR10P dataset \(0\.5% to 10% of the features\)\.Figure A166:ACC and AUC of the SVM classifier on the warpPIE10P dataset \(0\.5% to 10% of the features\)\.Figure A167:ACC and AUC of the SVM classifier on the Yale dataset \(0\.5% to 10% of the features\)\.Figure A168:ACC and AUC of the SVM classifier on the ALLAML dataset \(5% to 100% of the features\)\.Figure A169:ACC and AUC of the SVM classifier on the arcene dataset \(5% to 100% of the features\)\.Figure A170:ACC and AUC of the SVM classifier on the Carcinom dataset \(5% to 100% of the features\)\.Figure A171:ACC and AUC of the SVM classifier on the CLL\_SUB\_111 dataset \(5% to 100% of the features\)\.Figure A172:ACC and AUC of the SVM classifier on the COIL20 dataset \(5% to 100% of the features\)\.Figure A173:ACC and AUC of the SVM classifier on the colon dataset \(5% to 100% of the features\)\.Figure A174:ACC and AUC of the SVM classifier on the gisette dataset \(5% to 100% of the features\)\.Figure A175:ACC and AUC of the SVM classifier on the GLI\_85 dataset \(5% to 100% of the features\)\.Figure A176:ACC and AUC of the SVM classifier on the GLIOMA dataset \(5% to 100% of the features\)\.Figure A177:ACC and AUC of the SVM classifier on the Isolet dataset \(5% to 100% of the features\)\.Figure A178:ACC and AUC of the SVM classifier on the leukemia dataset \(5% to 100% of the features\)\.Figure A179:ACC and AUC of the SVM classifier on the lung dataset \(5% to 100% of the features\)\.Figure A180:ACC and AUC of the SVM classifier on the lung\_discrete dataset \(5% to 100% of the features\)\.Figure A181:ACC and AUC of the SVM classifier on the madelon dataset \(5% to 100% of the features\)\.Figure A182:ACC and AUC of the SVM classifier on the ORL dataset \(5% to 100% of the features\)\.Figure A183:ACC and AUC of the SVM classifier on the orlraws10P dataset \(5% to 100% of the features\)\.Figure A184:ACC and AUC of the SVM classifier on the pixraw10P dataset \(5% to 100% of the features\)\.Figure A185:ACC and AUC of the SVM classifier on the Prostate\_GE dataset \(5% to 100% of the features\)\.Figure A186:ACC and AUC of the SVM classifier on the SMK\_CAN\_187 dataset \(5% to 100% of the features\)\.Figure A187:ACC and AUC of the SVM classifier on the TOX\_171 dataset \(5% to 100% of the features\)\.Figure A188:ACC and AUC of the SVM classifier on the warpAR10P dataset \(5% to 100% of the features\)\.Figure A189:ACC and AUC of the SVM classifier on the warpPIE10P dataset \(5% to 100% of the features\)\.Figure A190:ACC and AUC of the SVM classifier on the Yale dataset \(5% to 100% of the features\)\.Figure A191:ACC and AUC of the random forest classifier on the ALLAML dataset \(0\.5% to 10% of the features\)\.Figure A192:ACC and AUC of the random forest classifier on the arcene dataset \(0\.5% to 10% of the features\)\.Figure A193:ACC and AUC of the random forest classifier on the Carcinom dataset \(0\.5% to 10% of the features\)\.Figure A194:ACC and AUC of the random forest classifier on the CLL\_SUB\_111 dataset \(0\.5% to 10% of the features\)\.Figure A195:ACC and AUC of the random forest classifier on the COIL20 dataset \(0\.5% to 10% of the features\)\.Figure A196:ACC and AUC of the random forest classifier on the colon dataset \(0\.5% to 10% of the features\)\.Figure A197:ACC and AUC of the random forest classifier on the gisette dataset \(0\.5% to 10% of the features\)\.Figure A198:ACC and AUC of the random forest classifier on the GLI\_85 dataset \(0\.5% to 10% of the features\)\.Figure A199:ACC and AUC of the random forest classifier on the GLIOMA dataset \(0\.5% to 10% of the features\)\.Figure A200:ACC and AUC of the random forest classifier on the leukemia dataset \(0\.5% to 10% of the features\)\.Figure A201:ACC and AUC of the random forest classifier on the lung dataset \(0\.5% to 10% of the features\)\.Figure A202:ACC and AUC of the random forest classifier on the lung\_discrete dataset \(0\.5% to 10% of the features\)\.Figure A203:ACC and AUC of the random forest classifier on the madelon dataset \(0\.5% to 10% of the features\)\.Figure A204:ACC and AUC of the random forest classifier on the ORL dataset \(0\.5% to 10% of the features\)\.Figure A205:ACC and AUC of the random forest classifier on the orlraws10P dataset \(0\.5% to 10% of the features\)\.Figure A206:ACC and AUC of the random forest classifier on the pixraw10P dataset \(0\.5% to 10% of the features\)\.Figure A207:ACC and AUC of the random forest classifier on the Prostate\_GE dataset \(0\.5% to 10% of the features\)\.Figure A208:ACC and AUC of the random forest classifier on the SMK\_CAN\_187 dataset \(0\.5% to 10% of the features\)\.Figure A209:ACC and AUC of the random forest classifier on the TOX\_171 dataset \(0\.5% to 10% of the features\)\.Figure A210:ACC and AUC of the random forest classifier on the warpAR10P dataset \(0\.5% to 10% of the features\)\.Figure A211:ACC and AUC of the random forest classifier on the warpPIE10P dataset \(0\.5% to 10% of the features\)\.Figure A212:ACC and AUC of the random forest classifier on the Yale dataset \(0\.5% to 10% of the features\)\.Figure A213:ACC and AUC of the random forest classifier on the ALLAML dataset \(5% to 100% of the features\)\.Figure A214:ACC and AUC of the random forest classifier on the arcene dataset \(5% to 100% of the features\)\.Figure A215:ACC and AUC of the random forest classifier on the Carcinom dataset \(5% to 100% of the features\)\.Figure A216:ACC and AUC of the random forest classifier on the CLL\_SUB\_111 dataset \(5% to 100% of the features\)\.Figure A217:ACC and AUC of the random forest classifier on the COIL20 dataset \(5% to 100% of the features\)\.Figure A218:ACC and AUC of the random forest classifier on the colon dataset \(5% to 100% of the features\)\.Figure A219:ACC and AUC of the random forest classifier on the gisette dataset \(5% to 100% of the features\)\.Figure A220:ACC and AUC of the random forest classifier on the GLI\_85 dataset \(5% to 100% of the features\)\.Figure A221:ACC and AUC of the random forest classifier on the GLIOMA dataset \(5% to 100% of the features\)\.Figure A222:ACC and AUC of the random forest classifier on the leukemia dataset \(5% to 100% of the features\)\.Figure A223:ACC and AUC of the random forest classifier on the lung dataset \(5% to 100% of the features\)\.Figure A224:ACC and AUC of the random forest classifier on the lung\_discrete dataset \(5% to 100% of the features\)\.Figure A225:ACC and AUC of the random forest classifier on the madelon dataset \(5% to 100% of the features\)\.Figure A226:ACC and AUC of the random forest classifier on the ORL dataset \(5% to 100% of the features\)\.Figure A227:ACC and AUC of the random forest classifier on the orlraws10P dataset \(5% to 100% of the features\)\.Figure A228:ACC and AUC of the random forest classifier on the pixraw10P dataset \(5% to 100% of the features\)\.Figure A229:ACC and AUC of the random forest classifier on the Prostate\_GE dataset \(5% to 100% of the features\)\.Figure A230:ACC and AUC of the random forest classifier on the SMK\_CAN\_187 dataset \(5% to 100% of the features\)\.Figure A231:ACC and AUC of the random forest classifier on the TOX\_171 dataset \(5% to 100% of the features\)\.Figure A232:ACC and AUC of the random forest classifier on the warpAR10P dataset \(5% to 100% of the features\)\.Figure A233:ACC and AUC of the random forest classifier on the warpPIE10P dataset \(5% to 100% of the features\)\.Figure A234:ACC and AUC of the random forest classifier on the Yale dataset \(5% to 100% of the features\)\.Figure A235:ACC and AUC of the XGBoost classifier on the ALLAML dataset \(0\.5% to 10% of the features\)\.Figure A236:ACC and AUC of the XGBoost classifier on the arcene dataset \(0\.5% to 10% of the features\)\.Figure A237:ACC and AUC of the XGBoost classifier on the Carcinom dataset \(0\.5% to 10% of the features\)\.Figure A238:ACC and AUC of the XGBoost classifier on the CLL\_SUB\_111 dataset \(0\.5% to 10% of the features\)\.Figure A239:ACC and AUC of the XGBoost classifier on the COIL20 dataset \(0\.5% to 10% of the features\)\.Figure A240:ACC and AUC of the XGBoost classifier on the colon dataset \(0\.5% to 10% of the features\)\.Figure A241:ACC and AUC of the XGBoost classifier on the gisette dataset \(0\.5% to 10% of the features\)\.Figure A242:ACC and AUC of the XGBoost classifier on the GLI\_85 dataset \(0\.5% to 10% of the features\)\.Figure A243:ACC and AUC of the XGBoost classifier on the GLIOMA dataset \(0\.5% to 10% of the features\)\.Figure A244:ACC and AUC of the XGBoost classifier on the Isolet dataset \(0\.5% to 10% of the features\)\.Figure A245:ACC and AUC of the XGBoost classifier on the leukemia dataset \(0\.5% to 10% of the features\)\.Figure A246:ACC and AUC of the XGBoost classifier on the lung dataset \(0\.5% to 10% of the features\)\.Figure A247:ACC and AUC of the XGBoost classifier on the lung\_discrete dataset \(0\.5% to 10% of the features\)\.Figure A248:ACC and AUC of the XGBoost classifier on the madelon dataset \(0\.5% to 10% of the features\)\.Figure A249:ACC and AUC of the XGBoost classifier on the ORL dataset \(0\.5% to 10% of the features\)\.Figure A250:ACC and AUC of the XGBoost classifier on the orlraws10P dataset \(0\.5% to 10% of the features\)\.Figure A251:ACC and AUC of the XGBoost classifier on the pixraw10P dataset \(0\.5% to 10% of the features\)\.Figure A252:ACC and AUC of the XGBoost classifier on the Prostate\_GE dataset \(0\.5% to 10% of the features\)\.Figure A253:ACC and AUC of the XGBoost classifier on the SMK\_CAN\_187 dataset \(0\.5% to 10% of the features\)\.Figure A254:ACC and AUC of the XGBoost classifier on the TOX\_171 dataset \(0\.5% to 10% of the features\)\.Figure A255:ACC and AUC of the XGBoost classifier on the warpAR10P dataset \(0\.5% to 10% of the features\)\.Figure A256:ACC and AUC of the XGBoost classifier on the warpPIE10P dataset \(0\.5% to 10% of the features\)\.Figure A257:ACC and AUC of the XGBoost classifier on the Yale dataset \(0\.5% to 10% of the features\)\.Figure A258:ACC and AUC of the XGBoost classifier on the ALLAML dataset \(5% to 100% of the features\)\.Figure A259:ACC and AUC of the XGBoost classifier on the arcene dataset \(5% to 100% of the features\)\.Figure A260:ACC and AUC of the XGBoost classifier on the Carcinom dataset \(5% to 100% of the features\)\.Figure A261:ACC and AUC of the XGBoost classifier on the CLL\_SUB\_111 dataset \(5% to 100% of the features\)\.Figure A262:ACC and AUC of the XGBoost classifier on the COIL20 dataset \(5% to 100% of the features\)\.Figure A263:ACC and AUC of the XGBoost classifier on the colon dataset \(5% to 100% of the features\)\.Figure A264:ACC and AUC of the XGBoost classifier on the gisette dataset \(5% to 100% of the features\)\.Figure A265:ACC and AUC of the XGBoost classifier on the GLI\_85 dataset \(5% to 100% of the features\)\.Figure A266:ACC and AUC of the XGBoost classifier on the GLIOMA dataset \(5% to 100% of the features\)\.Figure A267:ACC and AUC of the XGBoost classifier on the Isolet dataset \(5% to 100% of the features\)\.Figure A268:ACC and AUC of the XGBoost classifier on the leukemia dataset \(5% to 100% of the features\)\.Figure A269:ACC and AUC of the XGBoost classifier on the lung dataset \(5% to 100% of the features\)\.Figure A270:ACC and AUC of the XGBoost classifier on the lung\_discrete dataset \(5% to 100% of the features\)\.Figure A271:ACC and AUC of the XGBoost classifier on the madelon dataset \(5% to 100% of the features\)\.Figure A272:ACC and AUC of the XGBoost classifier on the ORL dataset \(5% to 100% of the features\)\.Figure A273:ACC and AUC of the XGBoost classifier on the orlraws10P dataset \(5% to 100% of the features\)\.Figure A274:ACC and AUC of the XGBoost classifier on the pixraw10P dataset \(5% to 100% of the features\)\.Figure A275:ACC and AUC of the XGBoost classifier on the Prostate\_GE dataset \(5% to 100% of the features\)\.Figure A276:ACC and AUC of the XGBoost classifier on the SMK\_CAN\_187 dataset \(5% to 100% of the features\)\.Figure A277:ACC and AUC of the XGBoost classifier on the TOX\_171 dataset \(5% to 100% of the features\)\.Figure A278:ACC and AUC of the XGBoost classifier on the warpAR10P dataset \(5% to 100% of the features\)\.Figure A279:ACC and AUC of the XGBoost classifier on the warpPIE10P dataset \(5% to 100% of the features\)\.Figure A280:ACC and AUC of the XGBoost classifier on the Yale dataset \(5% to 100% of the features\)\.