An evolutionary algorithm that constructs recurrent neural networks

Standard methods for simultaneously inducing the structure and weights of recurrent neural networks limit every task to an assumed class of architectures. Such a simplification is necessary since the interactions between network structure and function are not well understood. Evolutionary computations, which include genetic algorithms and evolutionary programming, are population-based search methods that have shown promise in many similarly complex tasks. This paper argues that genetic algorithms are inappropriate for network acquisition and describes an evolutionary program, called GNARL, that simultaneously acquires both the structure and weights for recurrent networks. GNARL's empirical acquisition method allows for the emergence of complex behaviors and topologies that are potentially excluded by the artificial architectural constraints imposed in standard network induction methods.

[1]  Lawrence J. Fogel,et al.  Artificial Intelligence through Simulated Evolution , 1966 .

[2]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[3]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[4]  Terrence J. Sejnowski,et al.  Parallel Networks that Learn to Pronounce English Text , 1987, Complex Syst..

[5]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[6]  Michael C. Mozer,et al.  Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment , 1988, NIPS.

[7]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[8]  T. Ash,et al.  Dynamic node creation in backpropagation networks , 1989, International 1989 Joint Conference on Neural Networks.

[9]  C. Lee Giles,et al.  Higher Order Recurrent Networks and Grammatical Inference , 1989, NIPS.

[10]  David E. Goldberg,et al.  Genetic Algorithms and Walsh Functions: Part I, A Gentle Introduction , 1989, Complex Syst..

[11]  David E. Goldberg,et al.  Genetic Algorithms and Walsh Functions: Part II, Deception and Its Analysis , 1989, Complex Syst..

[12]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[13]  Peter M. Todd,et al.  Designing Neural Networks using Genetic Algorithms , 1989, ICGA.

[14]  Stephen Jose Hanson,et al.  Meiosis Networks , 1989, NIPS.

[15]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[16]  L. Darrell Whitley,et al.  Genetic algorithms and neural networks: optimizing connections and connectivity , 1990, Parallel Comput..

[17]  Ronald J. Williams,et al.  Adaptive state representation and estimation using recurrent connectionist networks , 1990 .

[18]  Geoffrey E. Hinton,et al.  Distributed Representations , 1986, The Philosophy of Artificial Intelligence.

[19]  Andrew G. Barto,et al.  Connectionist learning for control , 1990 .

[20]  Scott E. Fahlman,et al.  The Recurrent Cascade-Correlation Architecture , 1990, NIPS.

[21]  Richard K. Belew,et al.  Evolving networks: using the genetic algorithm with connectionist learning , 1990 .

[22]  Marcus Frean,et al.  The Upstart Algorithm: A Method for Constructing and Training Feedforward Neural Networks , 1990, Neural Computation.

[23]  David R. Jefferson,et al.  An Artificial Neural Network Representation for Artificial Organisms , 1990, PPSN.

[24]  Raymond L. Watrous,et al.  Induction of Finite-State Automata Using Second-Order Recurrent Networks , 1991, NIPS.

[25]  John R. Koza,et al.  Genetic evolution and co-evolution of computer programs , 1991 .

[26]  C. Lee Giles,et al.  Extracting and Learning an Unknown Grammar with Recurrent Neural Networks , 1991, NIPS.

[27]  A. P. Wieland,et al.  Evolving neural network controllers for unstable systems , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[28]  John R. Koza,et al.  Genetic generation of both the weights and architecture for a neural network , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[29]  Jan Torreele,et al.  Temporal Processing with Recurrent Networks: An Evolutionary Approach , 1991, ICGA.

[30]  Darrell Whitley,et al.  Genetic cascade learning for neural networks , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[31]  Mitchell A. Potter,et al.  A genetic cascade-correlation learning algorithm , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[32]  Babak Hassibi,et al.  Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , 1992, NIPS.

[33]  J. D. Schaffer,et al.  Combinations of genetic algorithms and neural networks: a survey of the state of the art , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[34]  Randall D. Beer,et al.  Evolving Dynamical Neural Networks for Adaptive Behavior , 1992, Adapt. Behav..

[35]  J. R. McDonnell,et al.  Determining neural network connectivity using evolutionary programming , 1992, [1992] Conference Record of the Twenty-Sixth Asilomar Conference on Signals, Systems & Computers.

[36]  Padhraic Smyth,et al.  Learning Finite State Machines With Self-Clustering Recurrent Networks , 1993, Neural Computation.

[37]  Peter J. Angeline,et al.  Competitive Environments Evolve Better Solutions for Complex Tasks , 1993, ICGA.

[38]  Mahmood R. Azimi-Sadjadi,et al.  Recursive dynamic node creation in multilayer neural networks , 1993, IEEE Trans. Neural Networks.

[39]  D. B. Fogel,et al.  Using evolutionary programing to create neural networks that are capable of playing tic-tac-toe , 1993, IEEE International Conference on Neural Networks.

[40]  C. L. Giles,et al.  Constructive learning of recurrent neural networks , 1993, IEEE International Conference on Neural Networks.

[41]  C. Lee Giles,et al.  Pruning recurrent neural networks for improved generalization performance , 1994, IEEE Trans. Neural Networks.

[42]  David B. Fogel,et al.  An introduction to simulated evolutionary optimization , 1994, IEEE Trans. Neural Networks.

[43]  Giovanna Castellano,et al.  Pruning in Recurrent Neural Networks , 1994 .

[44]  Richard S. Sutton,et al.  Connectionist Learning for Control , 1995 .