A unified evolutionary training scheme for single and ensemble of feedforward neural network

Artificial neural networks (ANNs) have been successfully applied to many areas. The key of success is to properly tune the architecture and the connection weights of the ANN and to have the trained ANN own higher generalization ability. For complicated problems, artificial neural network ensemble classifier, instead of a single ANN classifier, is considered. But, it is not so straight forward to construct the ANN ensemble. In this paper, we propose a unified evolutionary training scheme (UETS) which can either train a generalized feedforward neural network or construct an ANN ensemble. The performance of the UETS was evaluated by applying it to solve the n-bit parity problem and the classification problems on five datasets from the UCI machine learning repository. By comparing with the previous studies, the experimental results reveal that the neural networks and the ensembles trained by the UETS have very good classification ability for unseen cases.

[1]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[2]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[3]  Martin A. Riedmiller,et al.  A direct adaptive method for faster backpropagation learning: the RPROP algorithm , 1993, IEEE International Conference on Neural Networks.

[4]  Shiro Usui,et al.  Mutation-based genetic neural network , 2005, IEEE Transactions on Neural Networks.

[5]  Sung-Bae Cho,et al.  Evolutionary ensemble of diverse artificial neural networks using speciation , 2008, Neurocomputing.

[6]  Victor O. K. Li,et al.  Evolutionary artificial neural network based on Chemical Reaction Optimization , 2011, 2011 IEEE Congress of Evolutionary Computation (CEC).

[7]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.

[8]  Xin Yao,et al.  Simultaneous training of negatively correlated neural networks in an ensemble , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[9]  Qun Dai,et al.  A novel ensemble pruning algorithm based on randomized greedy selective strategy and ballot , 2013, Neurocomputing.

[10]  Wen-Ching Chen,et al.  An evolutionary algorithm for variable-sized ANN ensemble construction , 2010, 2010 IEEE International Conference on Systems, Man and Cybernetics.

[11]  Tung-Kuan Liu,et al.  Tuning the structure and parameters of a neural network by using hybrid Taguchi-genetic algorithm , 2006, IEEE Trans. Neural Networks.

[12]  Huanhuan Chen,et al.  Multiobjective Neural Network Ensembles Based on Regularized Negative Correlation Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[13]  Teresa Bernarda Ludermir,et al.  An Optimization Methodology for Neural Network Weights and Architectures , 2006, IEEE Transactions on Neural Networks.

[14]  Xin Yao,et al.  Evolving artificial neural network ensembles , 2008 .

[15]  Athanasios Tsakonas,et al.  A fuzzy evolutionary framework for combining ensembles , 2013, Appl. Soft Comput..

[16]  Xin Yao,et al.  Ensemble learning via negative correlation , 1999, Neural Networks.

[17]  Chen Lin,et al.  LibD3C: Ensemble classifiers with a clustering and dynamic selection strategy , 2014, Neurocomputing.

[18]  Chia-Feng Juang,et al.  A hybrid of genetic algorithm and particle swarm optimization for recurrent network design , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[19]  K. Chou,et al.  Cell-PLoc: a package of Web servers for predicting subcellular localization of proteins in various organisms , 2008, Nature Protocols.

[20]  Hitoshi Iba,et al.  Learning polynomial feedforward neural networks by genetic programming and backpropagation , 2003, IEEE Trans. Neural Networks.

[21]  Nikola K. Kasabov,et al.  Fast neural network ensemble learning via negative-correlation data correction , 2005, IEEE Transactions on Neural Networks.

[22]  Xin Yao,et al.  Diversity creation methods: a survey and categorisation , 2004, Inf. Fusion.

[23]  Yuping Wang,et al.  An orthogonal genetic algorithm with quantization for global numerical optimization , 2001, IEEE Trans. Evol. Comput..

[24]  Huanhuan Chen,et al.  Evolutionary random neural ensembles based on negative correlation learning , 2007, 2007 IEEE Congress on Evolutionary Computation.

[25]  Wei Tang,et al.  Ensembling neural networks: Many could be better than all , 2002, Artif. Intell..

[26]  K. Chou,et al.  Prediction of protein structural classes. , 1995, Critical reviews in biochemistry and molecular biology.

[27]  Qun Dai,et al.  ModEnPBT: A Modified Backtracking Ensemble Pruning algorithm , 2013, Appl. Soft Comput..

[28]  Jacek M. Zurada,et al.  Artificial Intelligence and Soft Computing: 12th International Conference, ICAISC 2013, Zakopane, Poland, June 9-13, 2013, Proceedings, Part I ... / Lecture Notes in Artificial Intelligence) , 2013 .

[29]  Kuo-Chen Chou,et al.  Notes & TipsArtificial Neural Network Model for Predicting α-Turn Types☆ , 1999 .

[30]  S. Selvakumar,et al.  Detection of distributed denial of service attacks using an ensemble of adaptive and hybrid neuro-fuzzy systems , 2013, Comput. Commun..

[31]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[32]  Wen-Ching Chen,et al.  A Two-Phase Genetic Local Search Algorithm for Feedforward Neural Network Training , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[33]  Antoniya Georgieva,et al.  Neural Network Learning With Global Heuristic Search , 2007, IEEE Transactions on Neural Networks.

[34]  H.-B. Shen,et al.  Using ensemble classifier to identify membrane protein types , 2006, Amino Acids.

[35]  Nikola K. Kasabov,et al.  A Preliminary Study on Negative Correlation Learning via Correlation-Corrected Data (NCCD) , 2005, Neural Processing Letters.

[36]  Teresa Bernarda Ludermir,et al.  Clustering and co-evolution to construct neural network ensembles: An experimental study , 2008, Neural Networks.

[37]  K. Chou,et al.  Hum-PLoc: a novel ensemble classifier for predicting human protein subcellular localization. , 2006, Biochemical and biophysical research communications.

[38]  Fan Yang,et al.  Exploring the diversity in cluster ensemble generation: Random sampling and random projection , 2014, Expert Syst. Appl..

[39]  Paulo Cortez,et al.  Particle swarms for feedforward neural network training , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[40]  Chee Peng Lim,et al.  A multi-objective evolutionary algorithm-based ensemble optimizer for feature selection and classification with neural network models , 2014, Neurocomputing.

[41]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Lifeng Xi,et al.  Evolving artificial neural networks using an improved PSO and DPSO , 2008, Neurocomputing.

[43]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[44]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[45]  Yong Liu,et al.  Generate Different Neural Networks by Negative Correlation Learning , 2005, ICNC.

[46]  Mitsuo Gen,et al.  Genetic algorithms and engineering design , 1997 .

[47]  Antoniya Georgieva,et al.  Supervised Neural Network Training with a Hybrid Global Optimization Technique , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[48]  Xin Yao,et al.  Making use of population information in evolutionary artificial neural networks , 1998, IEEE Trans. Syst. Man Cybern. Part B.

[49]  Rudy Setiono,et al.  Use of a quasi-Newton method in a feedforward neural network construction algorithm , 1995, IEEE Trans. Neural Networks.

[50]  Hak-Keung Lam,et al.  Tuning of the structure and parameters of a neural network using an improved genetic algorithm , 2003, IEEE Trans. Neural Networks.

[51]  D. Opitz,et al.  Popular Ensemble Methods: An Empirical Study , 1999, J. Artif. Intell. Res..

[52]  Scott E. Fahlman,et al.  An empirical study of learning speed in back-propagation networks , 1988 .

[53]  Dianhui Wang,et al.  Evolutionary extreme learning machine ensembles with size control , 2013, Neurocomputing.

[54]  Tung-Kuan Liu,et al.  Hybrid Taguchi-genetic algorithm for global numerical optimization , 2004, IEEE Transactions on Evolutionary Computation.

[55]  Minqiang Li,et al.  Coevolutionary learning of neural network ensemble for complex classification tasks , 2012, Pattern Recognit..

[56]  Q. Henry Wu,et al.  Group Search Optimizer: An Optimization Algorithm Inspired by Animal Searching Behavior , 2009, IEEE Transactions on Evolutionary Computation.

[57]  Lars Kai Hansen,et al.  Neural Network Ensembles , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[58]  Amanda J. C. Sharkey,et al.  On Combining Artificial Neural Nets , 1996, Connect. Sci..

[59]  Dianhui Wang,et al.  Fast decorrelated neural network ensembles with random weights , 2014, Inf. Sci..

[60]  Xin Yao,et al.  Evolutionary ensembles with negative correlation learning , 2000, IEEE Trans. Evol. Comput..

[61]  Marcin Korytkowski,et al.  Negative Correlation Learning of Neuro-fuzzy System Ensembles , 2010, ICAISC.