Making use of population information in evolutionary artificial neural networks

This paper is concerned with the simultaneous evolution of artificial neural network (ANN) architectures and weights. The current practice in evolving ANN's is to choose the best ANN in the last generation as the final result. This paper proposes a different approach to form the final result by combining all the individuals in the last generation in order to make best use of all the information contained in the whole population. This approach regards a population of ANN's as an ensemble and uses a combination method to integrate them. Although there has been some work on integrating ANN modules, little has been done in evolutionary learning to make best use of its population information. Four linear combination methods have been investigated in this paper to illustrate our ideas. Three real-world data sets have been used in our experimental studies, which show that the recursive least-square (RLS) algorithm always produces an integrated system that outperforms the best individual. The results confirm that a population contains more information than a single individual. Evolutionary learning should exploit such information to improve generalization of learned systems.

[1]  E. M. Hartwell.  Boston , 1906 .

[2]  Lawrence J. Fogel,et al.  Artificial Intelligence through Simulated Evolution , 1966 .

[3]  H. Akaike.  A new look at the statistical model identification , 1974 .

[4]  Jack Belzer,et al.  Encyclopedia of Computer Science and Technology , 2002 .

[5]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[6]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[7]  C. Cowan,et al.  Adaptive Filters and Equalisers , 1988 .

[8]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[9]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[10]  D. B. Fogel,et al.  AN INFORMATION CRITERION FOR OPTIMAL NEURAL NETWORK SELECTION , 1990, 1990 Conference Record Twenty-Fourth Asilomar Conference on Signals, Systems and Computers, 1990..

[11]  Gilbert Syswerda,et al.  A Study of Reproduction in Generational and Steady State Genetic Algorithms , 1990, FOGA.

[12]  Richard K. Belew,et al.  Evolving networks: using the genetic algorithm with connectionist learning , 1990 .

[13]  David H. Wolpert,et al.  A Mathematical Theory of Generalization: Part II , 1990, Complex Syst..

[14]  David B. Fogel.  An information criterion for optimal neural network selection , 1991, IEEE Trans. Neural Networks.

[15]  Farid U. Dowla,et al.  Backpropagation Learning for Multilayer Feed-Forward Neural Networks Using the Conjugate Gradient Method , 1991, Int. J. Neural Syst..

[16]  Peter J. B. Hancock,et al.  Genetic algorithms and permutation problems: a comparison of recombination operators for neural net structure specification , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[17]  J. D. Schaffer,et al.  Combinations of genetic algorithms and neural networks: a survey of the state of the art , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[18]  Martin Fodslette Møller,et al.  A scaled conjugate gradient algorithm for fast supervised learning , 1993, Neural Networks.

[19]  Byoung-Tak Zhang,et al.  Evolving Optimal Neural Networks Using Genetic Algorithms with Occam's Razor , 1993, Complex Syst..

[20]  Dušan Petrovački,et al.  Evolutional development of a multilevel neural network , 1993, Neural Networks.

[21]  D.R. Hush,et al.  Progress in supervised neural networks , 1993, IEEE Signal Processing Magazine.

[22]  M. Perrone.  Improving regression estimation: Averaging methods for variance reduction with extensions to general convex measure optimization , 1993 .

[23]  Ferdinand Hergert,et al.  Improving model selection by nonconvergent methods , 1993, Neural Networks.

[24]  Xin Yao,et al.  A review of evolutionary artificial neural networks , 1993, Int. J. Intell. Syst..

[25]  Xin Yao,et al.  An empirical study of genetic operators in genetic algorithms , 1993, Microprocess. Microprogramming.

[26]  Lutz Prechelt,et al.  A Set of Neural Network Benchmark Problems and Benchmarking Rules , 1994 .

[27]  David J. Spiegelhalter,et al.  Machine Learning, Neural and Statistical Classification , 2009 .

[28]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[29]  Kurt Hornik,et al.  Learning in linear neural networks: a survey , 1995, IEEE Trans. Neural Networks.

[30]  Asim Roy,et al.  An algorithm to generate radial basis function (RBF)-like nets for classification problems , 1995, Neural Networks.

[31]  David B. Fogel,et al.  Evolutionary Computation: Towards a New Philosophy of Machine Intelligence , 1995 .

[32]  Xin Yao,et al.  A dilemma for fitness sharing with a scaling function , 1995, Proceedings of 1995 IEEE International Conference on Evolutionary Computation.

[33]  David B. Fogel,et al.  CONTINUOUS EVOLUTIONARY PROGRAMMING: ANALYSIS AND EXPERIMENTS , 1995 .

[34]  Xin Yao,et al.  A Preliminary Study on Designing Artiicial Neural Networks Using Co-evolution , 1995 .

[35]  Xin Yao,et al.  Every Niching Method has its Niche: Fitness Sharing and Implicit Sharing Compared , 1996, PPSN.

[36]  Xin YaoComputational.  A Population-Based Learning Algorithm Which Learns BothArchitectures and Weights of Neural Networks , 1996 .

[37]  Xin Yao,et al.  Automatic modularization by speciation , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[38]  Xin Yao,et al.  Ensemble structure of evolutionary artificial neural networks , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[39]  Sherif Hashem,et al.  Optimal Linear Combinations of Neural Networks , 1997, Neural Networks.

[40]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.

[41]  Thomas Bäck,et al.  Evolutionary computation: Toward a new philosophy of machine intelligence , 1997, Complex..

[42]  Xin Yao,et al.  Towards designing artificial neural networks by evolution , 1998 .