Coevolutionary learning of neural network ensemble for complex classification tasks

Ensemble approaches to classification have attracted a great deal of interest recently. This paper presents a novel method for designing the neural network ensemble using coevolutionary algorithm. The bootstrap resampling procedure is employed to obtain different training subsets that are used to estimate different component networks of the ensemble. Then the cooperative coevolutionary algorithm is developed to optimize the ensemble model via the divide-and-cooperative mechanism. All component networks are coevolved in parallel in the scheme of interacting co-adapted subpopulations. The fitness of an individual from a particular subpopulation is assessed by associating it with the representatives from other subpopulations. In order to promote the cooperation of all component networks, the proposed method considers both the accuracy and the diversity among the component networks that are evaluated using the multi-objective Pareto optimality measure. A hybrid output-combination method is designed to determine the final ensemble output. Experimental results illustrate that the proposed method is able to obtain neural network ensemble models with better classification accuracy in comparison with currently popular ensemble algorithms.

[1]  Oleksandr Makeyev,et al.  Neural network with ensembles , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[2]  Pedro M. Domingos.  A Unifeid Bias-Variance Decomposition and its Applications , 2000, ICML.

[3]  Minqiang Li,et al.  Improving multiclass pattern recognition with a co-evolutionary RBFNN , 2008, Pattern Recognit. Lett..

[4]  R. Schapire.  The Strength of Weak Learnability , 1990, Machine Learning.

[5]  Lars Kai Hansen,et al.  Neural Network Ensembles , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  Paul W. Munro,et al.  Competition Among Networks Improves Committee Performance , 1996, NIPS.

[7]  Lothar Thiele,et al.  Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach , 1999, IEEE Trans. Evol. Comput..

[8]  N. Garc'ia-Pedrajas,et al.  CIXL2: A Crossover Operator for Evolutionary Algorithms Based on Population Features , 2005, J. Artif. Intell. Res..

[9]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[10]  Ron Kohavi,et al.  Bias Plus Variance Decomposition for Zero-One Loss Functions , 1996, ICML.

[11]  Malcolm I. Heywood,et al.  Training genetic programming on half a million patterns: an example from anomaly detection , 2005, IEEE Transactions on Evolutionary Computation.

[12]  Xin Yao,et al.  Making use of population information in evolutionary artificial neural networks , 1998, IEEE Trans. Syst. Man Cybern. Part B.

[13]  Pedro M. Domingos.  A Unifeid Bias-Variance Decomposition and its Applications , 2000, ICML.

[14]  Xin Yao,et al.  Evolving artificial neural network ensembles , 2008, IEEE Computational Intelligence Magazine.

[15]  David Casasent,et al.  Radial basis function neural networks for nonlinear Fisher discrimination and Neyman-Pearson classification , 2003, Neural Networks.

[16]  Zhao Wei-xiang.  RBFN Structure Determination Strategy Based on PLS and GAs , 2002 .

[17]  Eric Bauer,et al.  An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants , 1999, Machine Learning.

[18]  Xin Yao,et al.  Evolutionary ensembles with negative correlation learning , 2000, IEEE Trans. Evol. Comput..

[19]  Tin Kam Ho,et al.  The Random Subspace Method for Constructing Decision Forests , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  Michael R. Berthold,et al.  Boosting the Performance of RBF Networks with Dynamic Decay Adjustment , 1994, NIPS.

[21]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[22]  Giandomenico Spezzano,et al.  Ensemble Techniques for Parallel Genetic Programming Based Classifiers , 2003, EuroGP.

[23]  Kenneth A. De Jong,et al.  Cooperative Coevolution: An Architecture for Evolving Coadapted Subcomponents , 2000, Evolutionary Computation.

[24]  Lawrence O. Hall,et al.  A scalable framework for cluster ensembles , 2009, Pattern Recognit..

[26]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[27]  Gonzalo Martínez-Muñoz,et al.  Switching class labels to generate classification ensembles , 2005, Pattern Recognit..

[28]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[29]  Xin Yao,et al.  Ensemble Learning Using Multi-Objective Evolutionary Algorithms , 2006, J. Math. Model. Algorithms.

[30]  Chen Fu-zan.  GA-RBFNN learning algorithm for complex classifications , 2006 .

[31]  César Hervás-Martínez,et al.  Cooperative coevolution of artificial neural network ensembles for pattern classification , 2005, IEEE Transactions on Evolutionary Computation.

[32]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[33]  Anders Krogh,et al.  Neural Network Ensembles, Cross Validation, and Active Learning , 1994, NIPS.

[34]  Jordan B. Pollack,et al.  Pareto Optimality in Coevolutionary Learning , 2001, ECAL.

[35]  Xin Yao,et al.  A constructive algorithm for training cooperative neural network ensembles , 2003, IEEE Trans. Neural Networks.