Genetic rule selection with a multi-classifier coding scheme for ensemble classifier design

In this paper, we examine the effectiveness of genetic rule selection with a multi-classifier coding scheme for ensemble classifier design. Genetic rule selection is a two-stage method. The first stage is rule extraction from numerical data using a data mining technique. Extracted rules are used as candidate rules. The second stage is evolutionary multiobjective rule selection from the candidate rules. We use a multi-classifier coding scheme where an ensemble classifier is represented by an integer string. Three criteria are used as objective functions in evolutionary multiobjective rule selection to optimize ensemble classifiers in terms of accuracy and diversity. We examine the performance of designed ensemble classifiers through computational experiments on six benchmark datasets in the UCI machine learning repository.

[1]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[2]  Hisao Ishibuchi,et al.  Evolutionary Multiobjective Knowledge Extraction for High-Dimensional Pattern Classification Problems , 2004, PPSN.

[3]  Hisao Ishibuchi,et al.  Fuzzy rule selection by multi-objective genetic local search algorithms and rule evaluation measures in data mining , 2004, Fuzzy Sets Syst..

[4]  D. Opitz,et al.  Popular Ensemble Methods: An Empirical Study , 1999, J. Artif. Intell. Res..

[5]  Hisao Ishibuchi,et al.  Comparison of Heuristic Criteria for Fuzzy Rule Selection in Classification Problems , 2004, Fuzzy Optim. Decis. Mak..

[6]  Hisao Ishibuchi,et al.  Effects of Three-Objective Genetic Rule Selection on the Generalization Ability of Fuzzy Rule-Based Systems , 2003, EMO.

[7]  G DietterichThomas.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees , 2000 .

[8]  Hisao Ishibuchi,et al.  Selecting fuzzy if-then rules for classification problems using genetic algorithms , 1995, IEEE Trans. Fuzzy Syst..

[9]  Bernhard Sendhoff,et al.  EVOLUTIONARY MULTI-OBJECTIVE OPTIMIZATION APPROACH TO CONSTRUCTING NEURAL NETWORK ENSEMBLES FOR REGRESSION , 2004 .

[10]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[11]  Eric Bauer,et al.  An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants , 1999, Machine Learning.

[12]  Xin Yao,et al.  An analysis of diversity measures , 2006, Machine Learning.

[13]  Hisao Ishibuchi,et al.  Single-objective and two-objective genetic algorithms for selecting linguistic rules for pattern classification problems , 1997, Fuzzy Sets Syst..

[14]  Xin Yao,et al.  Evolutionary ensembles with negative correlation learning , 2000, IEEE Trans. Evol. Comput..

[15]  Tapio Elomaa,et al.  General and Efficient Multisplitting of Numerical Attributes , 1999, Machine Learning.

[16]  Hisao Ishibuchi,et al.  Rule weight specification in fuzzy rule-based classification systems , 2005, IEEE Transactions on Fuzzy Systems.

[17]  Hisao Ishibuchi,et al.  Classification and modeling with linguistic information granules - advanced approaches to linguistic data mining , 2004, Advanced information processing.

[18]  Toshiharu Hatanaka,et al.  Pattern Classification by Evolutionary RBF Networks Ensemble Based on Multi-objective Optimization , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[19]  Hisao Ishibuchi,et al.  Three-objective genetics-based machine learning for linguistic rule extraction , 2001, Inf. Sci..

[20]  Xin Yao,et al.  Evolutionary framework for the construction of diverse hybrid ensembles , 2005, ESANN.

[21]  Gary B. Lamont,et al.  Evolutionary Algorithms for Solving Multi-Objective Problems (Genetic and Evolutionary Computation) , 2006 .

[22]  Luiz Eduardo Soares de Oliveira,et al.  Multi-objective Genetic Algorithms to Create Ensemble of Classifiers , 2005, EMO.

[23]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[24]  Luiz Eduardo Soares de Oliveira,et al.  Feature selection for ensembles:a hierarchical multi-objective genetic algorithm approach , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[25]  Hussein A. Abbass,et al.  Pareto neuro-evolution: constructing ensemble of neural networks using multi-objective optimization , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[26]  Mykola Pechenizkiy,et al.  Diversity in search strategies for ensemble feature selection , 2005, Inf. Fusion.

[27]  Antonio González Muñoz,et al.  SLAVE: a genetic learning system based on an iterative approach , 1999, IEEE Trans. Fuzzy Syst..

[28]  Hisao Ishibuchi,et al.  Evolutionary Multiobjective Optimization for Generating an Ensemble of Fuzzy Rule-Based Classifiers , 2003, GECCO.

[29]  Kalyanmoy Deb,et al.  Multi-objective optimization using evolutionary algorithms , 2001, Wiley-Interscience series in systems and optimization.

[30]  Ron Kohavi,et al.  Bias Plus Variance Decomposition for Zero-One Loss Functions , 1996, ICML.

[31]  Bernhard Sendhoff,et al.  Neural network regularization and ensembling using multi-objective evolutionary algorithms , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[32]  Gary B. Lamont,et al.  Applications Of Multi-Objective Evolutionary Algorithms , 2004 .

[33]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[34]  Thomas G. Dietterich.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization , 2000, Machine Learning.

[35]  Xin Yao,et al.  DIVACE: Diverse and Accurate Ensemble Learning Algorithm , 2004, IDEAL.

[36]  Hisao Ishibuchi,et al.  Designing Fuzzy Ensemble Classifiers by Evolutionary Multiobjective Optimization with an Entropy-Based Diversity Criterion , 2006, 2006 Sixth International Conference on Hybrid Intelligent Systems (HIS'06).

[37]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1997, EuroCOLT.