Evolutionary ensemble of diverse artificial neural networks using speciation

Recently, many researchers have designed neural network architectures with evolutionary algorithms but most of them have used only the fittest solution of the last generation. To better exploit information, an ensemble of individuals is a more promising choice because information that is derived from combining a set of classifiers might produce higher accuracy than merely using the information from the best classifier among them. One of the major factors for optimum accuracy is the diversity of the classifier set. In this paper, we present a method of generating diverse evolutionary neural networks through fitness sharing and then combining these networks by the behavior knowledge space method. Fitness sharing that shares resources if the distance between the individuals is smaller than the sharing radius is a representative speciation method, which produces diverse results than standard evolutionary algorithms that converge to only one solution. Especially, the proposed method calculates the distance between the individuals using average output, Pearson correlation and modified Kullback-Leibler entropy to enhance fitness sharing performance. In experiments with Australian credit card assessment, breast cancer, and diabetes in the UCI database, the proposed method performed better than not only the non-speciation method but also better than previously published methods.

[1]  Adam Krzyżak,et al.  Methods of combining multiple classifiers and their applications to handwriting recognition , 1992, IEEE Trans. Syst. Man Cybern..

[2]  X. Yao Evolving Artificial Neural Networks , 1999 .

[3]  Xin Yao,et al.  Learning and Evolution by Minimization of Mutual Information , 2002, PPSN.

[4]  Louis A. Tamburino,et al.  E-Net: Evolutionary neural network synthesis , 2002, Neurocomputing.

[5]  Joon-Hyun Ahn,et al.  Speciated neural networks evolved with fitness sharing technique , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[6]  Xin Yao,et al.  Speciation Techniques in Evolved Ensembles with Negative Correlation Learning , 2006, 2006 IEEE International Conference on Evolutionary Computation.

[7]  Richard Lippmann,et al.  Neural Network Classifiers Estimate Bayesian a posteriori Probabilities , 1991, Neural Computation.

[8]  Stephen I. Gallant,et al.  Perceptron-based learning algorithms , 1990, IEEE Trans. Neural Networks.

[9]  Fuad Rahman,et al.  Multiple classifier decision combination strategies for character recognition: A review , 2003, Document Analysis and Recognition.

[10]  David E. Goldberg,et al.  Genetic Algorithms with Sharing for Multimodalfunction Optimization , 1987, ICGA.

[11]  Shun-ichi Amari,et al.  Information geometry of the EM and em algorithms for neural networks , 1995, Neural Networks.

[12]  Xin Yao,et al.  Evolving Neural Network Ensembles by Fitness Sharing , 2006, 2006 IEEE International Conference on Evolutionary Computation.

[13]  Richard P. Lippmann,et al.  Neural Networks, Bayesian a posteriori Probabilities, and Pattern Classification , 1994 .

[14]  Hussein A. Abbass,et al.  An evolutionary artificial neural networks approach for breast cancer diagnosis , 2002, Artif. Intell. Medicine.

[15]  Rudy Setiono,et al.  Use of a quasi-Newton method in a feedforward neural network construction algorithm , 1995, IEEE Trans. Neural Networks.

[16]  Seung-Ik Lee,et al.  Exploiting diversity of neural ensembles with speciated evolution , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[17]  David Corne,et al.  Predicting alarms in supermarket refrigeration systems using evolved neural networks and evolved rulesets , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[18]  Xin Yao,et al.  Diversity creation methods: a survey and categorisation , 2004, Inf. Fusion.

[19]  Xin Yao,et al.  Making use of population information in evolutionary artificial neural networks , 1998, IEEE Trans. Syst. Man Cybern. Part B.

[20]  Pedro Ángel Castillo Valdivieso,et al.  G-Prop-II: global optimization of multilayer perceptrons using GAs , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[21]  Xin Yao,et al.  Evolutionary ensembles with negative correlation learning , 2000, IEEE Trans. Evol. Comput..

[22]  Hualou Liang,et al.  Improvement of Cascade Correlation Learning Algorithm with an Evolutionary Initialization , 1998, Inf. Sci..

[23]  Harry Wechsler,et al.  From Statistics to Neural Networks: Theory and Pattern Recognition Applications , 1996 .

[24]  Sung-Bae Cho,et al.  Fuzzy integration of structure adaptive SOMs for web content mining , 2004, Fuzzy Sets Syst..

[25]  Sung-Bae Cho,et al.  Effective Fingerprint Classification by Localized Models of Support Vector Machines , 2006, ICB.

[26]  Yu-Bin Yang,et al.  Lung cancer cell identification based on artificial neural network ensembles , 2002, Artif. Intell. Medicine.

[27]  Mübeccel Demirekler,et al.  Speaker identification by combining multiple classifiers using Dempster-Shafer theory of evidence , 2003, Speech Commun..

[28]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[29]  César Hervás-Martínez,et al.  COVNET: a cooperative coevolutionary model for evolving artificial neural networks , 2003, IEEE Trans. Neural Networks.

[30]  G. W. Milligan,et al.  Methodology Review: Clustering Methods , 1987 .

[31]  Terry Windeatt,et al.  Diversity measures for multiple classifier system analysis and design , 2004, Inf. Fusion.

[32]  H Qian Relative entropy: free energy associated with equilibrium fluctuations and nonequilibrium deviations. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[33]  Paul J. Werbos,et al.  The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting , 1994 .

[34]  A. Khotanzad,et al.  Hand written digit recognition using BKS combination of neural network classifiers , 1994, Proceedings of the IEEE Southwest Symposium on Image Analysis and Interpretation.

[35]  R. Mojena,et al.  Hierarchical Grouping Methods and Stopping Rules: An Evaluation , 1977, Comput. J..

[36]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[37]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[38]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[39]  Luís A. Alexandre,et al.  On combining classifiers using sum and product rules , 2001, Pattern Recognit. Lett..

[40]  Francis K. H. Quek,et al.  Attribute bagging: improving accuracy of classifier ensembles by using random feature subsets , 2003, Pattern Recognit..

[41]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.

[42]  Xin Yao,et al.  Towards designing artificial neural networks by evolution , 1998 .

[43]  Xin Yao,et al.  Simultaneous training of negatively correlated neural networks in an ensemble , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[44]  Adam Prügel-Bennett,et al.  Genetic drift in genetic algorithm selection schemes , 1999, IEEE Trans. Evol. Comput..

[45]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .