Simultaneous instance and feature selection and weighting using evolutionary computation: Proposal and study

HighlightsThe paper presents a framework for simultaneous instance and feature selection and weighting.An extensive comparison of all the possible combinations is carried out.The proposal is also studied in class-imbalanced datasets. Current research is constantly producing an enormous amount of information, which presents a challenge for data mining algorithms. Many of the problems in some of the most relevant research areas, such as bioinformatics, security and intrusion detection or text mining, involve large or huge datasets. Data mining algorithms are seriously challenged by these datasets. One of the most common methods to handle large datasets is data reduction. Among others, feature and instance selection are arguably the most commonly used methods for data reduction. Conversely, feature and instance weighting focus on improving the performance of the data mining task.Due to the different aims of these four methods, instance and feature selection and weighting, they can be combined to improve the performance of the data mining methods used. In this paper, a general framework for combining these four tasks is presented, and a comprehensive study of the usefulness of the 15 possible combinations is performed.Using a large set of 80 problems, a study of the behavior of all possible combinations in classification performance, data reduction and execution time is carried out. These factors are also studied using 60 class-imbalanced datasets.

[1]  Shih-Fu Chang,et al.  Image Retrieval: Current Techniques, Promising Directions, and Open Issues , 1999, J. Vis. Commun. Image Represent..

[2]  Enrique Vidal,et al.  Learning weighted metrics to minimize nearest-neighbor classification error , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Jörg Kindermann,et al.  Text Categorization with Support Vector Machines. How to Represent Texts in Input Space? , 2002, Machine Learning.

[4]  Hiroshi Motoda,et al.  Feature Selection for Knowledge Discovery and Data Mining , 1998, The Springer International Series in Engineering and Computer Science.

[5]  Salvatore J. Stolfo,et al.  Adaptive Intrusion Detection: A Data Mining Approach , 2000, Artificial Intelligence Review.

[6]  Pedro Larrañaga,et al.  Prototype Selection and Feature Subset Selection by Estimation of Distribution Algorithms. A Case Study in the Survival of Cirrhotic Patients Treated with TIPS , 2001, AIME.

[7]  Huan Liu,et al.  Feature selection for clustering - a filter solution , 2002, 2002 IEEE International Conference on Data Mining, 2002. Proceedings..

[8]  Shih-Wei Lin,et al.  Parameter tuning, feature selection and weight assignment of features for case-based reasoning by artificial immune system , 2011, Appl. Soft Comput..

[9]  J. T. de Souza,et al.  A novel approach for integrating feature and instance selection , 2008, ICMLC 2008.

[10]  Hans-Peter Kriegel,et al.  Feature Weighting and Instance Selection for Collaborative Filtering: An Information-Theoretic Approach* , 2003, Knowledge and Information Systems.

[11]  C. A. Murthy,et al.  Unsupervised Feature Selection Using Feature Similarity , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Javier Pérez-Rodríguez,et al.  A scalable approach to simultaneous evolutionary instance and feature selection , 2013, Inf. Sci..

[13]  Fernando Fernández,et al.  Local Feature Weighting in Nearest Prototype Classification , 2008, IEEE Transactions on Neural Networks.

[14]  Larry J. Eshelman,et al.  The CHC Adaptive Search Algorithm: How to Have Safe Search When Engaging in Nontraditional Genetic Recombination , 1990, FOGA.

[15]  R. Leardi,et al.  Genetic algorithms applied to feature selection in PLS regression: how and when to use them , 1998 .

[16]  Ee-Peng Lim,et al.  On strategies for imbalanced text classification using SVM: A comparative study , 2009, Decis. Support Syst..

[17]  Javier Pérez-Rodríguez,et al.  OligoIS: Scalable Instance Selection for Class-Imbalanced Data Sets , 2013, IEEE Transactions on Cybernetics.

[18]  Zexuan Zhu,et al.  Wrapper–Filter Feature Selection Algorithm Using a Memetic Framework , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[19]  Francisco Herrera,et al.  Using evolutionary algorithms as instance selection for data reduction in KDD: an experimental study , 2003, IEEE Trans. Evol. Comput..

[20]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[21]  Shengrui Wang,et al.  Particle swarm optimizer for variable weighting in clustering high-dimensional data , 2009, 2009 IEEE Swarm Intelligence Symposium.

[22]  Zhaohong Deng,et al.  Robust Relief-Feature Weighting, Margin Maximization, and Fuzzy Optimization , 2010, IEEE Transactions on Fuzzy Systems.

[23]  Francisco Herrera,et al.  Prototype Selection for Nearest Neighbor Classification: Taxonomy and Empirical Study , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Chris Mellish,et al.  Advances in Instance Selection for Instance-Based Learning Algorithms , 2002, Data Mining and Knowledge Discovery.

[25]  Jiye Liang,et al.  A novel attribute weighting algorithm for clustering high-dimensional categorical data , 2011, Pattern Recognit..

[26]  Nicolás García-Pedrajas,et al.  Large scale instance selection by means of federal instance selection , 2012, Data Knowl. Eng..

[27]  Francisco Herrera,et al.  Stratification for scaling up evolutionary prototype selection , 2005, Pattern Recognit. Lett..

[28]  Krzysztof Krawiec,et al.  Evolutionary weighting of image features for diagnosing of CNS tumors , 2000, Artif. Intell. Medicine.

[29]  Michael I. Jordan,et al.  Feature selection for high-dimensional genomic microarray data , 2001, ICML.

[30]  V. Alarcon-Aquino,et al.  Instance Selection and Feature Weighting Using Evolutionary Algorithms , 2006, 2006 15th International Conference on Computing.

[31]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[32]  Huan Liu,et al.  Discretization: An Enabling Technique , 2002, Data Mining and Knowledge Discovery.

[33]  Carla E. Brodley,et al.  Recursive automatic bias selection for classifier construction , 1995, Machine Learning.

[34]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[35]  Pat Langley,et al.  Selection of Relevant Features and Examples in Machine Learning , 1997, Artif. Intell..

[36]  Hung-Ming Chen,et al.  Design of nearest neighbor classifiers: multi-objective approach , 2005, Int. J. Approx. Reason..

[37]  Javier Pérez-Rodríguez,et al.  Multi-selection of instances: A straightforward way to improve evolutionary instance selection , 2012, Appl. Soft Comput..

[38]  Nicolás García-Pedrajas,et al.  Boosting for class-imbalanced datasets using genetically evolved supervised non-linear projections , 2012, Progress in Artificial Intelligence.

[39]  Paul Scheunders,et al.  Genetic feature selection combined with composite fuzzy nearest neighbor classifiers for hyperspectral satellite imagery , 2002, Pattern Recognit. Lett..

[40]  Lakhmi C. Jain,et al.  Nearest neighbor classifier: Simultaneous editing and feature selection , 1999, Pattern Recognit. Lett..

[41]  Pierre Gançarski,et al.  Comparison between two coevolutionary feature weighting algorithms in clustering , 2008, Pattern Recognit..

[42]  Xin Yao,et al.  Evolving a cooperative population of neural networks by minimizing mutual information , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[43]  T. Warren Liao,et al.  Classification of weld flaws with imbalanced class data , 2008, Expert Syst. Appl..

[44]  Jun Zhou,et al.  MILIS: Multiple Instance Learning with Instance Selection , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  David W. Aha,et al.  A Review and Empirical Evaluation of Feature Weighting Methods for a Class of Lazy Learning Algorithms , 1997, Artificial Intelligence Review.

[46]  Foster J. Provost,et al.  A Survey of Methods for Scaling Up Inductive Algorithms , 1999, Data Mining and Knowledge Discovery.

[47]  Ahmed Bouridane,et al.  Simultaneous feature selection and feature weighting using Hybrid Tabu Search/K-nearest neighbor classifier , 2007, Pattern Recognit. Lett..

[48]  Hisao Ishibuchi,et al.  Pattern and Feature Selection by Genetic Algorithms in Nearest Neighbor Classification , 2000, Journal of Advanced Computational Intelligence and Intelligent Informatics.

[49]  Serkawt Khola,et al.  Feature Weighting and Selection - A Novel Genetic Evolutionary Approach , 2011 .

[50]  Nicolás García-Pedrajas,et al.  Constructing Ensembles of Classifiers by Means of Weighted Instance Selection , 2009, IEEE Transactions on Neural Networks.

[51]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[52]  Huan Liu,et al.  On Issues of Instance Selection , 2002, Data Mining and Knowledge Discovery.

[53]  E. Delgado,et al.  Feature weighting and selection using a hybrid approach based on Rademacher complexity model selection , 2007, 2007 Computers in Cardiology.

[54]  M. Friedman.  A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings , 1940 .

[55]  Nicolás García-Pedrajas,et al.  A cooperative coevolutionary algorithm for instance selection for instance-based learning , 2010, Machine Learning.

[56]  Keinosuke Fukunaga,et al.  A Branch and Bound Algorithm for Feature Subset Selection , 1977, IEEE Transactions on Computers.

[57]  Nicolás García-Pedrajas,et al.  Democratic instance selection: A linear complexity instance selection algorithm based on classifier ensemble concepts , 2010, Artif. Intell..

[58]  Wei Liu,et al.  Class Confidence Weighted kNN Algorithms for Imbalanced Data Sets , 2011, PAKDD.

[59]  Yvan Saeys,et al.  Translation initiation site prediction on a genomic scale: beauty in simplicity , 2007, ISMB/ECCB.

[60]  Huan Liu,et al.  Customer Retention via Data Mining , 2000, Artificial Intelligence Review.

[61]  Huan Liu,et al.  Toward integrating feature selection algorithms for classification and clustering , 2005, IEEE Transactions on Knowledge and Data Engineering.

[62]  Francisco Herrera,et al.  Integrating Instance Selection, Instance Weighting, and Feature Weighting for Nearest Neighbor Classifiers by Coevolutionary Algorithms , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[63]  Francisco Herrera,et al.  IFS-CoCo: Instance and feature selection based on cooperative coevolution with nearest neighbor rule , 2010, Pattern Recognit..

[64]  J. Friedman.  Special Invited Paper-Additive logistic regression: A statistical view of boosting , 2000 .

[65]  Tony R. Martinez,et al.  Weighted Instance Typicality Search (WITS): A nearest neighbor data reduction algorithm , 2004, Intell. Data Anal..

[66]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[67]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[68]  Kai Ming Ting,et al.  An Instance-weighting Method to Induce Cost-sensitive Trees , 2001 .

[69]  Sung-Bae Cho,et al.  Efficient huge-scale feature selection with speciated genetic algorithm , 2005 .

[70]  Jack Sklansky,et al.  A note on genetic algorithms for large-scale feature selection , 1989, Pattern Recognit. Lett..

[71]  Huan Liu,et al.  Feature Selection for Classification , 1997, Intell. Data Anal..