A genetic algorithm to obtain the optimal recurrent neural network

Abstract Selecting the optimal topology of a neural network for a particular application is a difficult task. In the case of recurrent neural networks, most methods only induce topologies in which their neurons are fully connected. In this paper, we present a genetic algorithm capable of obtaining not only the optimal topology of a recurrent neural network but also the least number of connections necessary. Finally, this genetic algorithm is applied to a problem of grammatical inference using neural networks, with very good results.

[1]  David Beasley,et al.  An overview of genetic algorithms: Part 1 , 1993 .

[2]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[3]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[4]  Ah Chung Tsoi,et al.  Rule inference for financial prediction using recurrent neural networks , 1997, Proceedings of the IEEE/IAFE 1997 Computational Intelligence for Financial Engineering (CIFEr).

[5]  C. Lee Giles,et al.  Pruning recurrent neural networks for improved generalization performance , 1994, IEEE Trans. Neural Networks.

[6]  Larry J. Eshelman,et al.  Preventing Premature Convergence in Genetic Algorithms by Preventing Incest , 1991, ICGA.

[7]  C. Lee Giles,et al.  Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks , 1992, Neural Computation.

[8]  Sandiway Fong,et al.  Natural language grammatical inference: a comparison of recurrent neural networks and machine learning methods , 1995, Learning for Natural Language Processing.

[9]  M. W. Shields.  An Introduction to Automata Theory , 1988 .

[10]  James L. McClelland,et al.  Finite State Automata and Simple Recurrent Networks , 1989, Neural Computation.

[11]  Darrell Whitley,et al.  A genetic algorithm tutorial , 1994, Statistics and Computing.

[12]  S. A. Solla.  Capacity control in classifiers for pattern recognition , 1992, Neural Networks for Signal Processing II Proceedings of the 1992 IEEE Workshop.

[13]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[14]  Hava T. Siegelmann,et al.  The complexity of language recognition by neural networks , 1992, Neurocomputing.

[15]  Carl H. Smith,et al.  Inductive Inference: Theory and Methods , 1983, CSUR.

[16]  D. Searls,et al.  A syntactic pattern recognition system for DNA sequences , 1993 .

[17]  C. L. Giles,et al.  Second-order recurrent neural networks for grammatical inference , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[18]  J. Galletly.  An Overview of Genetic Algorithms , 1992 .

[19]  Bull,et al.  An Overview of Genetic Algorithms: Part 2, Research Topics , 1993 .

[20]  David B. Beasley,et al.  An overview of genetic algorithms: Part 1 , 1993 .

[21]  C. Lee Giles,et al.  Experimental Comparison of the Effect of Order in Recurrent Neural Networks , 1993, Int. J. Pattern Recognit. Artif. Intell..

[22]  Michael C. Mozer,et al.  Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment , 1988, NIPS.

[23]  C. Lee Giles,et al.  What Size Neural Network Gives Optimal Generalization? Convergence Properties of Backpropagation , 1998 .

[24]  C. Lee Giles,et al.  Constructive learning of recurrent neural networks: limitations of recurrent cascade correlation and a simple solution , 1995, IEEE Trans. Neural Networks.

[25]  Darrell Whitley,et al.  Genitor: a different genetic algorithm , 1988 .

[26]  C. L. Giles,et al.  Inserting rules into recurrent neural networks , 1992, Neural Networks for Signal Processing II Proceedings of the 1992 IEEE Workshop.

[27]  Larry J. Eshelman.  The CHC Adaptive Search Algo-rithm , 1991 .

[28]  Padhraic Smyth,et al.  Learning Finite State Machines With Self-Clustering Recurrent Networks , 1993, Neural Computation.

[29]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1996, Springer Berlin Heidelberg.

[30]  O. Firschein,et al.  Syntactic pattern recognition and applications , 1983, Proceedings of the IEEE.

[31]  Michael C. Mozer,et al.  Using Relevance to Reduce Network Size Automatically , 1989 .

[32]  Hikmet Senay,et al.  Fuzzy command grammars for intelligent interface design , 1992, IEEE Trans. Syst. Man Cybern..

[33]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[34]  David B. Searls.  Representing Genetic Information with Formal Grammars , 1988, AAAI.

[35]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[36]  Padhraic Smyth,et al.  Discrete recurrent neural networks for grammatical inference , 1994, IEEE Trans. Neural Networks.

[37]  Scott E. Fahlman,et al.  The Recurrent Cascade-Correlation Architecture , 1990, NIPS.

[38]  Man-Wai Mak,et al.  Exploring the effects of Lamarckian and Baldwinian learning in evolving recurrent neural networks , 1997, Proceedings of 1997 IEEE International Conference on Evolutionary Computation (ICEC '97).