An evolutionary approach to training feedforward and recurrent neural networks

This paper describes a method of utilising genetic algorithms to train fixed architecture feedforward and recurrent neural networks. The technique described uses the genetic algorithm to evolve changes to the weights and biases of the network rather than the weights and biases themselves. Results achieved by this technique indicate that for many problems it compares very favourably with the more common gradient descent techniques for training neural networks, and in some cases is superior. The technique is useful for those problem which are known to be difficult for the gradient descent techniques.

[1]  W. Spears,et al.  On the Virtues of Parameterized Uniform Crossover , 1995 .

[2]  Risto Miikkulainen,et al.  Evolving Complex Othello Strategies Using Marker-Based Genetic Encoding ofNeural Networks , 1993 .

[3]  Peter C. McCluskey,et al.  Feedforward and Recurrent Neural Networks and Genetic Programs for Stock and Time Series Forecasting , 1993 .

[4]  Richard K. Belew,et al.  Evolving networks: using the genetic algorithm with connectionist learning , 1990 .

[5]  David E. Goldberg,et al.  Real-coded Genetic Algorithms, Virtual Alphabets, and Blocking , 1991, Complex Syst..

[6]  J. Elman.  Distributed Representations, Simple Recurrent Networks, And Grammatical Structure , 1991 .

[7]  Paul J. Werbos,et al.  The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting , 1994 .

[8]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[9]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[10]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[11]  Ken Sharman,et al.  Evolving Recurrent Neural Network Architectures by Genetic Programming , 1996 .

[12]  Risto Miikkulainen,et al.  Hierarchical evolution of neural networks , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[13]  Melanie Mitchell,et al.  An introduction to genetic algorithms , 1996 .

[14]  Vasant Honavar,et al.  Evolutionary Design of Neural Architectures -- A Preliminary Taxonomy and Guide to Literature , 1995 .

[15]  L. Darrell Whitley,et al.  Genetic Reinforcement Learning with Multilayer Neural Networks , 1991, ICGA.

[16]  Nicol N. Schraudolph,et al.  A User's Guide to GAucsd 1.4 , 1992 .

[17]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[18]  Mitchell A. Potter,et al.  A genetic cascade-correlation learning algorithm , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[19]  A.I. Esparcia-Alcazar,et al.  Genetic programming techniques that evolve recurrent neural network architectures for signal processing , 1996, Neural Networks for Signal Processing VI. Proceedings of the 1996 IEEE Signal Processing Society Workshop.

[20]  Scott E. Fahlman,et al.  An empirical study of learning speed in back-propagation networks , 1988 .

[21]  Peter M. Todd,et al.  Designing Neural Networks using Genetic Algorithms , 1989, ICGA.

[22]  Frank Rosenblatt,et al.  PRINCIPLES OF NEURODYNAMICS. PERCEPTRONS AND THE THEORY OF BRAIN MECHANISMS , 1963 .

[23]  Peter G. Korning,et al.  Training neural networks by means of genetic algorithms working on very long chromosomes , 1995, Int. J. Neural Syst..

[24]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[25]  Nicholas J. Radcliffe,et al.  Genetic neural networks on MIMD computers , 1992 .

[26]  G. Kane.  Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol 1: Foundations, vol 2: Psychological and Biological Models , 1994 .

[27]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[28]  Rajendra Krishnan,et al.  2DELTA-GANN: A NEW APPROACH TO TRAINING NEURAL NETWORKS USING GENETIC ALGORITHMS , 1994 .

[29]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[30]  L. Darrell Whitley,et al.  Optimizing Neural Networks Using FasterMore Accurate Genetic Search , 1989, ICGA.

[31]  Warren S. Sarle,et al.  Stopped Training and Other Remedies for Overfitting , 1995 .

[32]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[33]  Michael I. Jordan.  Attractor dynamics and parallelism in a connectionist sequential machine , 1990 .

[34]  Fernando J. Pineda,et al.  Dynamics and architecture for neural computation , 1988, J. Complex..

[35]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[36]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[37]  José Nelson Amaral,et al.  Designing genetic algorithms for the state assignment problem , 1995, IEEE Trans. Syst. Man Cybern..

[38]  Brad Fullmer and Risto Miikkulainen.  Using Marker-Based Genetic Encoding Of Neural Networks To Evolve Finite-State Behaviour , 1991 .

[39]  D. E. Goldberg,et al.  Genetic Algorithms in Search, Optimization & Machine Learning , 1989 .

[40]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.

[41]  Rich Caruana,et al.  Removing the Genetics from the Standard Genetic Algorithm , 1995, ICML.

[42]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[43]  James L. McClelland,et al.  Explorations in parallel distributed processing: a handbook of models, programs, and exercises , 1988 .

[44]  David E. Moriarty,et al.  Symbiotic Evolution of Neural Networks in Sequential Decision Tasks , 1997 .

[45]  Donald R. Tveter.  The pattern recognition basis of artificial intelligence , 1998 .

[46]  Lawrence J. Fogel,et al.  Artificial Intelligence through Simulated Evolution , 1966 .