Multi‐objective ensemble generation

Ensemble methods that combine a committee of machine‐learning models, each known as a member or base learner, have gained research interests in the past decade. One interest on ensemble generation involves the multi‐objective approach, which attempts to generate both accurate and diverse members that fulfill the theoretical requirements of good ensembles. These methods resolve common difficulties of balancing the trade‐off between accuracy and diversity and have been shown to be advantageous over single‐objective methods. This study presents an up‐to‐date survey on multi‐objective ensemble generation methods, including widely used diversity measures, member generation, selection, and integration techniques. Challenges and potential applications of multi‐objective ensemble generation are also discussed. WIREs Data Mining Knowl Discov 2015, 5:234–245. doi: 10.1002/widm.1158

[1]  G. Yule.  On the Association of Attributes in Statistics: With Illustrations from the Material of the Childhood Society, &c , 1900 .

[2]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[3]  Luiz Eduardo Soares de Oliveira,et al.  Multi-objective Genetic Algorithms to Create Ensemble of Classifiers , 2005, EMO.

[4]  Robert P. W. Duin,et al.  An experimental study on diversity for bagging and boosting with linear classifiers , 2002, Inf. Fusion.

[5]  Yaochu Jin,et al.  Evolutionary multi-objective generation of recurrent neural network ensembles for time series prediction , 2014, Neurocomputing.

[6]  K D Wernecke,et al.  A coupling procedure for the discrimination of mixed data. , 1992, Biometrics.

[7]  Xin Yao,et al.  Ensemble Learning Using Multi-Objective Evolutionary Algorithms , 2006, J. Math. Model. Algorithms.

[8]  Peter J. Fleming,et al.  Many-Objective Optimization: An Engineering Design Perspective , 2005, EMO.

[9]  Stefan C. Kremer,et al.  Clustering unlabeled data with SOMs improves classification of labeled real-world data , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[10]  Yaochu Jin,et al.  Neural network ensembles for image identification using Pareto-optimal features , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[11]  James C. Bezdek,et al.  Decision templates for multiple classifier fusion: an experimental comparison , 2001, Pattern Recognit..

[12]  Xin Yao,et al.  A multi-objective ensemble method for online class imbalance learning , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[13]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[14]  Bernhard Sendhoff,et al.  A Multiobjective Evolutionary Algorithm Using Gaussian Process-Based Inverse Modeling , 2015, IEEE Transactions on Evolutionary Computation.

[15]  Peter J. Fleming,et al.  Genetic Algorithms for Multiobjective Optimization: FormulationDiscussion and Generalization , 1993, ICGA.

[16]  B. Everitt,et al.  Statistical methods for rates and proportions , 1973 .

[17]  Qingfu Zhang,et al.  Multiobjective evolutionary algorithms: A survey of the state of the art , 2011, Swarm Evol. Comput..

[18]  Robert Sabourin,et al.  A dynamic overproduce-and-choose strategy for the selection of classifier ensembles , 2008, Pattern Recognit..

[19]  Hussein A. Abbass,et al.  Pareto neuro-evolution: constructing ensemble of neural networks using multi-objective optimization , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[20]  Nigel M. Allinson,et al.  Fast committee learning: preliminary results , 1998 .

[21]  G DietterichThomas.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees , 2000 .

[22]  Ron Kohavi,et al.  Bias Plus Variance Decomposition for Zero-One Loss Functions , 1996, ICML.

[23]  Marco Laumanns,et al.  SPEA2: Improving the strength pareto evolutionary algorithm , 2001 .

[24]  Luiz Eduardo Soares de Oliveira,et al.  Feature Selection for Ensembles Using the Multi-Objective Optimization Approach , 2006, Multi-Objective Machine Learning.

[25]  Concha Bielza,et al.  Multiobjective Estimation of Distribution Algorithm Based on Joint Modeling of Objectives and Variables , 2014, IEEE Transactions on Evolutionary Computation.

[26]  Mykola Pechenizkiy,et al.  Diversity in search strategies for ensemble feature selection , 2005, Inf. Fusion.

[27]  Mark Johnston,et al.  Ensemble Learning and Pruning in Multi-Objective Genetic Programming for Classification with Unbalanced Data , 2011, Australasian Conference on Artificial Intelligence.

[28]  Hisao Ishibuchi,et al.  Indicator-based evolutionary algorithm with hypervolume approximation by achievement scalarizing functions , 2010, GECCO '10.

[29]  Kalyanmoy Deb,et al.  Muiltiobjective Optimization Using Nondominated Sorting in Genetic Algorithms , 1994, Evolutionary Computation.

[30]  Lucas Bradstreet,et al.  A Fast Way of Calculating Exact Hypervolumes , 2012, IEEE Transactions on Evolutionary Computation.

[31]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[32]  Oscar Cordón,et al.  Multiobjective genetic classifier selection for random oracles fuzzy rule-based classifier ensembles: How beneficial is the additional diversity? , 2013, Knowl. Based Syst..

[33]  D. Opitz,et al.  Popular Ensemble Methods: An Empirical Study , 1999, J. Artif. Intell. Res..

[34]  Hisao Ishibuchi,et al.  Simultaneous use of different scalarizing functions in MOEA/D , 2010, GECCO '10.

[35]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[36]  Lothar Thiele,et al.  The Hypervolume Indicator Revisited: On the Design of Pareto-compliant Indicators Via Weighted Integration , 2007, EMO.

[37]  Hussein A. Abbass,et al.  A Memetic Pareto Evolutionary Approach to Artificial Neural Networks , 2001, Australian Joint Conference on Artificial Intelligence.

[38]  David B. Skalak,et al.  The Sources of Increased Accuracy for Two Proposed Boosting Algorithms , 1996, AAAI 1996.

[39]  Xin Yao,et al.  An analysis of diversity measures , 2006, Machine Learning.

[40]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1997, EuroCOLT.

[41]  Robert Sabourin,et al.  Single and Multi-Objective Genetic Algorithms for the Selection of Ensemble of Classifiers , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[42]  Mitsuo Gen,et al.  Specification of Genetic Search Directions in Cellular Multi-objective Genetic Algorithms , 2001, EMO.

[43]  Ching Y. Suen,et al.  A Method of Combining Multiple Experts for the Recognition of Unconstrained Handwritten Numerals , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[44]  Robert Sabourin,et al.  Optimizing nearest neighbour in random subspaces using a multi-objective genetic algorithm , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[45]  Thomas G. Dietterich.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization , 2000, Machine Learning.

[46]  Derek Partridge,et al.  Diversity between Neural Networks and Decision Trees for Building Multiple Classifier Systems , 2000, Multiple Classifier Systems.

[47]  David H. Wolpert,et al.  Stacked generalization , 1992, Neural Networks.

[48]  Zhi-Hua Zhou,et al.  Multi-information Ensemble Diversity , 2010, MCS.

[49]  Lothar Thiele,et al.  Multiobjective Optimization Using Evolutionary Algorithms - A Comparative Case Study , 1998, PPSN.

[50]  Xin Yao,et al.  DIVACE: Diverse and Accurate Ensemble Learning Algorithm , 2004, IDEAL.

[51]  Padraig Cunningham,et al.  Using Diversity in Preparing Ensembles of Classifiers Based on Different Feature Subsets to Minimize Generalization Error , 2001, ECML.

[52]  Oleksandr Makeyev,et al.  Neural network with ensembles , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[53]  A. Sima Etaner-Uyar,et al.  Multiobjective evolutionary clustering of Web user sessions: a case study in Web page recommendation , 2010, Soft Comput..

[54]  Yaochu Jin,et al.  Generating diverse and accurate classifier ensembles using multi-objective optimization , 2014, 2014 IEEE Symposium on Computational Intelligence in Multi-Criteria Decision-Making (MCDM).

[55]  Martin J. Oates,et al.  The Pareto Envelope-Based Selection Algorithm for Multi-objective Optimisation , 2000, PPSN.

[56]  Frank Neumann,et al.  On the Effects of Adding Objectives to Plateau Functions , 2009, IEEE Transactions on Evolutionary Computation.

[57]  Bogdan Gabrys,et al.  Genetic algorithms in classifier fusion , 2006, Appl. Soft Comput..

[58]  H. Kita,et al.  Failure of Pareto-based MOEAs: does non-dominated really mean near to optimal? , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[59]  Gavin Brown,et al.  Ensemble Learning , 2010, Encyclopedia of Machine Learning and Data Mining.

[60]  Mark Johnston,et al.  Evolving ensembles in multi-objective genetic programming for classification with unbalanced data , 2011, GECCO '11.

[61]  Wei Tang,et al.  Ensembling neural networks: Many could be better than all , 2002, Artif. Intell..

[62]  Peter Tiño,et al.  Managing Diversity in Regression Ensembles , 2005, J. Mach. Learn. Res..

[63]  Xin Yao,et al.  Making use of population information in evolutionary artificial neural networks , 1998, IEEE Trans. Syst. Man Cybern. Part B.

[64]  Bernhard Sendhoff,et al.  Adapting Weighted Aggregation for Multiobjective Evolution Strategies , 2001, EMO.

[65]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[66]  Fabio Roli,et al.  Design of effective neural network ensembles for image classification purposes , 2001, Image Vis. Comput..

[67]  Jason Teo,et al.  Neural network ensembles for video game AI using evolutionary multi-objective optimization , 2011, 2011 11th International Conference on Hybrid Intelligent Systems (HIS).

[68]  Fabio Roli,et al.  Methods for Designing Multiple Classifier Systems , 2001, Multiple Classifier Systems.

[69]  Gerrit K. Janssens,et al.  Pareto-optimality of oblique decision trees from evolutionary algorithms , 2011, J. Glob. Optim..

[70]  Anders Krogh,et al.  Neural Network Ensembles, Cross Validation, and Active Learning , 1994, NIPS.

[71]  Hisao Ishibuchi,et al.  A multi-objective genetic local search algorithm and its application to flowshop scheduling , 1998, IEEE Trans. Syst. Man Cybern. Part C.

[72]  Qingfu Zhang,et al.  MOEA/D: A Multiobjective Evolutionary Algorithm Based on Decomposition , 2007, IEEE Transactions on Evolutionary Computation.

[73]  Xin Yao,et al.  Evolving hybrid ensembles of learning machines for better generalisation , 2006, Neurocomputing.

[74]  Robert Sabourin,et al.  Overfitting cautious selection of classifier ensembles with genetic algorithms , 2009, Inf. Fusion.

[75]  Bogdan Gabrys,et al.  Application of the Evolutionary Algorithms for Classifier Selection in Multiple Classifier Systems with Majority Voting , 2001, Multiple Classifier Systems.

[76]  Xin Yao,et al.  An analysis of multi-objective evolutionary algorithms for training ensemble models based on different performance measures in software effort estimation , 2013, PROMISE.

[77]  Gavin Brown.  An Information Theoretic Perspective on Multiple Classifier Systems , 2009, MCS.

[78]  Qingfu Zhang,et al.  This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION 1 RM-MEDA: A Regularity Model-Based Multiobjective Estimation of , 2022 .

[79]  Kagan Tumer,et al.  Error Correlation and Error Reduction in Ensemble Classifiers , 1996, Connect. Sci..

[80]  Nicola Beume,et al.  SMS-EMOA: Multiobjective selection based on dominated hypervolume , 2007, Eur. J. Oper. Res..

[81]  Amanda J. C. Sharkey,et al.  Multi-Net Systems , 1999 .

[82]  Christian Gagné,et al.  Evolutionary multiobjective optimization for selecting members of an ensemble streamflow forecasting model , 2013, GECCO '13.

[83]  Xin Yao,et al.  Ensemble learning via negative correlation , 1999, Neural Networks.

[84]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[85]  Martin J. Oates,et al.  PESA-II: region-based selection in evolutionary multiobjective optimization , 2001 .

[86]  Derek Partridge,et al.  Software Diversity: Practical Statistics for Its Measurement and Exploitation | Draft Currently under Revision , 1996 .

[87]  Xin Yao,et al.  Diversity creation methods: a survey and categorisation , 2004, Inf. Fusion.

[88]  Yang Yu,et al.  Diversity Regularized Ensemble Pruning , 2012, ECML/PKDD.

[89]  Bernhard Sendhoff,et al.  Pareto-Based Multiobjective Machine Learning: An Overview and Case Studies , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[90]  Bernhard Sendhoff,et al.  Neural network regularization and ensembling using multi-objective evolutionary algorithms , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[91]  Chee Peng Lim,et al.  A multi-objective evolutionary algorithm-based ensemble optimizer for feature selection and classification with neural network models , 2014, Neurocomputing.

[92]  David W. Opitz,et al.  Generating Accurate and Diverse Members of a Neural-Network Ensemble , 1995, NIPS.

[93]  Bogdan Gabrys,et al.  Classifier selection for majority voting , 2005, Inf. Fusion.

[94]  Robert P. W. Duin,et al.  Limits on the majority vote accuracy in classifier fusion , 2003, Pattern Analysis & Applications.

[95]  L. Cooper,et al.  When Networks Disagree: Ensemble Methods for Hybrid Neural Networks , 1992 .

[96]  Lothar Thiele,et al.  Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach , 1999, IEEE Trans. Evol. Comput..

[97]  Juan José Rodríguez Diez,et al.  Rotation Forest: A New Classifier Ensemble Method , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[98]  Ludmila I. Kuncheva,et al.  Relationships between combination methods and measures of diversity in combining classifiers , 2002, Inf. Fusion.

[99]  Rajib Mall,et al.  Application of elitist multi-objective genetic algorithm for classification rule generation , 2008, Appl. Soft Comput..

[100]  Nikunj C. Oza,et al.  Online Ensemble Learning , 2000, AAAI/IAAI.

[101]  Ludmila I. Kuncheva,et al.  Switching between selection and fusion in combining classifiers: an experiment , 2002, IEEE Trans. Syst. Man Cybern. Part B.