Selecting the best artificial neural network model from a multi-objective Differential Evolution Pareto front

The objective of this work is to select artificial neural network models (ANN) automatically with sigmoid basis units for multiclassification tasks. These models are designed using a Memetic Pareto Differential Evolution Neural Network algorithm (MPDENN) based on the Pareto dominance concept. We propose different methodologies to obtain the best model from the Pareto front obtained with the MPDENN algorithm. These methodologies are based on choosing the best models for training in both objectives, the Correct Classification Rate and Minimum Sensitivity, and the two models closest to the centroids of two clusters formed with the models of the first and second Pareto fronts. These methodologies are compared with three standard ensembles methodologies with very competitive results.

[1]  Sergios Theodoridis,et al.  Pattern Recognition, Third Edition , 2006 .

[2]  H. Abbass,et al.  PDE: a Pareto-frontier differential evolution approach for multi-objective optimization problems , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[3]  Xin Yao,et al.  A constructive algorithm for training cooperative neural network ensembles , 2003, IEEE Trans. Neural Networks.

[4]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[5]  A. Tamhane,et al.  Multiple Comparison Procedures , 2009 .

[6]  Joni-Kristian Kämäräinen,et al.  Differential Evolution Training Algorithm for Feed-Forward Neural Networks , 2003, Neural Processing Letters.

[7]  M. Friedman.  A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings , 1940 .

[8]  Henrik Boström,et al.  Ensemble member selection using multi-objective optimization , 2009, 2009 IEEE Symposium on Computational Intelligence and Data Mining.

[9]  Lior Rokach,et al.  Taxonomy for characterizing ensemble methods in classification tasks: A review and annotated bibliography , 2009, Comput. Stat. Data Anal..

[10]  Xin Yao,et al.  DIVACE: Diverse and Accurate Ensemble Learning Algorithm , 2004, IDEAL.

[11]  D. Opitz,et al.  Popular Ensemble Methods: An Empirical Study , 1999, J. Artif. Intell. Res..

[12]  Pedro Antonio Gutiérrez,et al.  Memetic Pareto Differential Evolution for Designing Artificial Neural Networks in Multiclassification Problems Using Cross-Entropy Versus Sensitivity , 2009, HAIS.

[13]  Hussein A. Abbass,et al.  An evolutionary artificial neural networks approach for breast cancer diagnosis , 2002, Artif. Intell. Medicine.

[14]  Huanhuan Chen,et al.  Multiobjective Neural Network Ensembles Based on Regularized Negative Correlation Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[15]  Guoqiang Peter Zhang,et al.  Neural networks for classification: a survey , 2000, IEEE Trans. Syst. Man Cybern. Part C.

[16]  Pedro Antonio Gutiérrez,et al.  Memetic Pareto Evolutionary Artificial Neural Networks to determine growth/no-growth in predictive microbiology , 2011, Appl. Soft Comput..

[17]  Christian Igel,et al.  Empirical evaluation of the improved Rprop learning algorithms , 2003, Neurocomputing.

[18]  Christian Igel,et al.  Improving the Rprop Learning Algorithm , 2000 .

[19]  Xin Yao,et al.  Ensemble learning via negative correlation , 1999, Neural Networks.

[20]  Rainer Storn,et al.  Differential Evolution Research – Trends and Open Questions , 2008 .

[21]  Huanhuan Chen,et al.  Regularized Negative Correlation Learning for Neural Network Ensembles , 2009, IEEE Transactions on Neural Networks.

[22]  R. Storn,et al.  Differential Evolution: A Practical Approach to Global Optimization (Natural Computing Series) , 2005 .

[23]  Xin Yao,et al.  Diversity creation methods: a survey and categorisation , 2004, Inf. Fusion.

[24]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[25]  Teresa Bernarda Ludermir,et al.  Clustering and co-evolution to construct neural network ensembles: An experimental study , 2008, Neural Networks.

[26]  César Hervás-Martínez,et al.  Memetic pareto differential evolutionary artificial neural networks to determine growth multi-classes in predictive microbiology , 2010, Evol. Intell..

[27]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[28]  Bernhard Sendhoff,et al.  Pareto-Based Multiobjective Machine Learning: An Overview and Case Studies , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[29]  Xin Yao,et al.  Ensemble Learning Using Multi-Objective Evolutionary Algorithms , 2006, J. Math. Model. Algorithms.

[30]  César Hervás-Martínez,et al.  Cooperative coevolution of artificial neural network ensembles for pattern classification , 2005, IEEE Transactions on Evolutionary Computation.

[31]  Xin Yao,et al.  Evolutionary ensembles with negative correlation learning , 2000, IEEE Trans. Evol. Comput..

[32]  Charles Gide,et al.  Cours d'économie politique , 1911 .

[33]  Pedro Antonio Gutiérrez,et al.  Sensitivity Versus Accuracy in Multiclass Problems Using Memetic Pareto Evolutionary Neural Networks , 2010, IEEE Transactions on Neural Networks.

[34]  Uday K. Chakraborty,et al.  Advances in Differential Evolution , 2010 .

[35]  Hussein A. Abbass,et al.  Pareto neuro-evolution: constructing ensemble of neural networks using multi-objective optimization , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[36]  Md. Zakirul Alam Bhuiyan.  An Algorithm for Determining Neural Network Architecture Using Differential Evolution , 2009, 2009 International Conference on Business Intelligence and Financial Engineering.