Bi-objective feature selection for discriminant analysis in two-class classification

This works deals with the problem of selecting variables (features) that are subsequently used in discriminant analysis. The aim is to find, from a set of m variables, smaller subsets which enable an efficient classification of cases in two classes. We consider two objectives, each one associated with the misclassification error in each class (type I and type II errors). Thus, we establish a bi-objective problem and develop an algorithm based on the NSGA-II strategy to this specific problem, in order to obtain a set of non-dominated solutions. Managing these two objectives separately (and not jointly) allows an enhanced analysis of the obtained solutions by observing the approach to efficient frontier. This is especially significant when each type of error has a different level of importance or when they cannot be compared. To illustrate these issues, several known databases from literature are used, as well as an additional database with several Spanish firms featured by financial variables and two classes: ''creditworthy'' and ''non-creditworthy''. Finally, we show that when solutions obtained by our NSGA-II implementation are evaluated from the classic mono-objective perspective (minimizing the ratio of both error types jointly) they are better than those obtained by classic methods for feature selection and similar than those provided by other recently published methods.

[1]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[2]  Kalyanmoy Deb,et al.  A Fast Elitist Non-dominated Sorting Genetic Algorithm for Multi-objective Optimisation: NSGA-II , 2000, PPSN.

[3]  Klaus Truemper,et al.  A method for controlling errors in two-class classification , 1999, Proceedings. Twenty-Third Annual International Computer Software and Applications Conference (Cat. No.99CB37032).

[4]  Belén Melián-Batista,et al.  Solving feature subset selection problem by a Parallel Scatter Search , 2006, Eur. J. Oper. Res..

[5]  Fakhri Karray,et al.  Multi-objective Feature Selection with NSGA II , 2007, ICANNGA.

[6]  Manuel Landajo,et al.  Bankruptcy prediction models based on multinorm analysis: An alternative to accounting ratios , 2012, Knowl. Based Syst..

[7]  Chih-Fong Tsai,et al.  Feature selection in bankruptcy prediction , 2009, Knowl. Based Syst..

[8]  Chih-Fong Tsai,et al.  Simple instance selection for bankruptcy prediction , 2012, Knowl. Based Syst..

[9]  Philip M. Lewis,et al.  The characteristic selection problem in recognition systems , 1962, IRE Trans. Inf. Theory.

[10]  Pedro Larrañaga,et al.  Feature Subset Selection by Bayesian network-based optimization , 2000, Artif. Intell..

[11]  Hung-Yi Lin,et al.  Feature selection based on cluster and variability analyses for ordinal multi-class classification problems , 2013, Knowl. Based Syst..

[12]  M. Tahar Kechadi,et al.  Multi-objective feature selection by using NSGA-II for customer churn prediction in telecommunications , 2010, Expert Syst. Appl..

[13]  Pedro Larrañaga,et al.  Prototype Selection and Feature Subset Selection by Estimation of Distribution Algorithms. A Case Study in the Survival of Cirrhotic Patients Treated with TIPS , 2001, AIME.

[14]  Young-Chan Lee,et al.  The random subspace binary logit (RSBL) model for bankruptcy prediction , 2011, Knowl. Based Syst..

[15]  R. Stolzenberg,et al.  Multiple Regression Analysis , 2004 .

[16]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[17]  Youfu Li,et al.  Multiple kernels for generalised discriminant analysis , 2010 .

[18]  Alper Ekrem Murat,et al.  A discrete particle swarm optimization method for feature selection in binary classification problems , 2010, Eur. J. Oper. Res..

[19]  George S. Sebestyen,et al.  Decision-making processes in pattern recognition , 1962 .

[20]  George S Sebestyen,et al.  Decision-making processes in pattern recognition (ACM monograph series) , 1962 .

[21]  C. Emmanouilidis,et al.  A multiobjective evolutionary setting for feature selection and a commonality-based crossover operator , 2000, Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512).

[22]  Qingfu Zhang,et al.  Multiobjective evolutionary algorithms: A survey of the state of the art , 2011, Swarm Evol. Comput..

[23]  Gang Wang,et al.  A novel bankruptcy prediction model based on an adaptive fuzzy k-nearest neighbor method , 2011, Knowl. Based Syst..

[24]  Ponnuthurai N. Suganthan,et al.  Feature Analysis and Classification of Protein Secondary Structure Data , 2003, ICANN.

[25]  Yong Wang,et al.  Incremental learning of complete linear discriminant analysis for face recognition , 2012, Knowl. Based Syst..

[26]  Ping Zhang,et al.  Supervised immune clonal evolutionary classification algorithm for high-dimensional data , 2012, Neurocomputing.

[27]  Joaquín A. Pacheco,et al.  A variable selection method based on Tabu search for logistic regression models , 2009, Eur. J. Oper. Res..

[28]  Lin Sun,et al.  Feature selection using rough entropy-based uncertainty measures in incomplete decision systems , 2012, Knowl. Based Syst..

[29]  Nasser Mozayani,et al.  A new multi-objective evolutionary approach for creating ensemble of classifiers , 2007, 2007 IEEE International Conference on Systems, Man and Cybernetics.

[30]  Jihoon Yang,et al.  Prediction of Molecular Bioactivity for Drug Design Using a Decision Tree Algorithm , 2003, Discovery Science.

[31]  Gary B. Lamont,et al.  Evolutionary Algorithms for Solving Multi-Objective Problems , 2002, Genetic Algorithms and Evolutionary Computation.

[32]  Enrico Zio,et al.  NSGA-II-trained neural network approach to the estimation of prediction intervals of scale deposition rate in oil & gas equipment , 2013, Expert Syst. Appl..

[33]  E. Zio,et al.  Application of a niched Pareto genetic algorithm for selecting features for nuclear transients classification , 2009 .

[34]  M. Fireman,et al.  MULTIPLE REGRESSION ANALYSIS OF SOIL DATA , 1954 .

[35]  Jiawei Han,et al.  Feature selection using dynamic weights for classification , 2013, Knowl. Based Syst..

[36]  Hiroshi Motoda,et al.  Feature Selection for Knowledge Discovery and Data Mining , 1998, The Springer International Series in Engineering and Computer Science.

[37]  Man Leung Wong,et al.  Data mining using parallel Multi-Objective Evolutionary algorithms on graphics hardware , 2010, IEEE Congress on Evolutionary Computation.

[38]  Joaquín A. Pacheco,et al.  Analysis of new variable selection methods for discriminant analysis , 2006, Comput. Stat. Data Anal..

[39]  Yang Zhang,et al.  A Generic Multi-dimensional Feature Extraction Method Using Multiobjective Genetic Programming , 2009, Evolutionary Computation.

[40]  Enrique Alba,et al.  Sensitivity and specificity based multiobjective approach for feature selection: Application to cancer diagnosis , 2009, Inf. Process. Lett..

[41]  Héctor Pomares,et al.  Parallel multiobjective memetic RBFNNs design and feature selection for function approximation problems , 2009, Neurocomputing.

[42]  V. J. Rayward-Smith,et al.  Data mining rules using multi-objective evolutionary algorithms , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[43]  Harald Ganster,et al.  Automated Melanoma Recognition , 2001, IEEE Trans. Medical Imaging.

[44]  M. Tada,et al.  Gene-Expression Profile Changes Correlated with Tumor Progression and Lymph Node Metastasis in Esophageal Cancer , 2004, Clinical Cancer Research.

[45]  C. J. Huberty,et al.  Applied Discriminant Analysis , 1994 .

[46]  P. Larra,et al.  Feature Subset Selection by Bayesian Networks Based Optimization Abstract|a New Method for Feature Subset Selection in Machine Learning, Fss-ebna , 1999 .

[47]  Francisco Herrera,et al.  A Multiobjective Evolutionary Approach to Concurrently Learn Rule and Data Bases of Linguistic Fuzzy-Rule-Based Systems , 2009, IEEE Transactions on Fuzzy Systems.