A Review of Evolutionary Artiicial Neural Networks 1 2

Research on potential interactions between connectionist learning systems, i.e., articial neural networks (ANNs), and evolutionary search procedures, like genetic algorithms (GAs), has attracted a lot of attention recently. Evolutionary ANNs (EANNs) can be considered as the combination of ANNs and evolutionary search procedures. This paper rst distinguishes among three kinds of evolution in EANNs, i.e., the evolution of connection weights, of architectures and of learning rules. Then it reviews each kind of evolution in detail and analyses critical issues related to di erent evolutions. The review shows that although a lot of work has been done on the evolution of connection weights and of architectures, few attempts have been made to understand the evolution of learning rules. Interactions among di erent evolutions are seldom mentioned in current research. However, the evolution of learning rules and its interactions with other kinds of evolution play a vital role in EANNs. As the nal part, this paper brie y describes a general framework for EANNs, which not only includes the aforementioned three kinds of evolution, but also considers interactions among them.

[1]  Lawrence J. Fogel,et al.  Artificial Intelligence through Simulated Evolution , 1966 .

[2]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[3]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[4]  B. Lewin.  Units of transcription and translation: Sequence components of heterogeneous nuclear RNA and messenger RNA , 1975, Cell.

[5]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[6]  Temple F. Smith.  Occam's razor , 1980, Nature.

[7]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[8]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[9]  M. Alexander,et al.  Principles of Neural Science , 1981 .

[10]  G. Edelman.  Neural Darwinism: The Theory Of Neuronal Group Selection , 1989 .

[11]  H. Szu,et al.  Nonconvex optimization by fast simulated annealing , 1987, Proceedings of the IEEE.

[12]  M. A. Rogers,et al.  Principles of Neural Science, 2nd ed , 1987 .

[13]  John Maynard Smith,et al.  When learning guides evolution , 1987, Nature.

[14]  J. David Schaffer,et al.  An Adaptive Crossover Distribution Mechanism for Genetic Algorithms , 1987, ICGA.

[15]  Terrence J. Sejnowski,et al.  Parallel Networks that Learn to Pronounce English Text , 1987, Complex Syst..

[16]  Aviv Bergman,et al.  BREEDING INTELLIGENT AUTOMATA. , 1987 .

[17]  Geoffrey E. Hinton,et al.  How Learning Can Guide Evolution , 1996, Complex Syst..

[18]  D. Ackley.  A connectionist machine for genetic hillclimbing , 1987 .

[19]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[20]  Terrence J. Sejnowski,et al.  Learned classification of sonar targets using a massively parallel network , 1988, IEEE Trans. Acoust. Speech Signal Process..

[21]  Michael C. Mozer,et al.  Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment , 1988, NIPS.

[22]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[23]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[24]  Yamashita,et al.  Backpropagation algorithm which varies the number of hidden units , 1989 .

[25]  M. V. Rossum,et al.  In Neural Computation , 2022 .

[26]  Thomas P. Caudell,et al.  Parametric Connectivity: Training of Constrained Networks using Genetic Algorithms , 1989, ICGA.

[27]  Tariq Samad,et al.  Towards the Genetic Synthesisof Neural Networks , 1989, ICGA.

[28]  L. Darrell Whitley,et al.  The GENITOR Algorithm and Selection Pressure: Why Rank-Based Allocation of Reproductive Trials is Best , 1989, ICGA.

[29]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[30]  L. Darrell Whitley,et al.  Optimizing Neural Networks Using FasterMore Accurate Genetic Search , 1989, ICGA.

[31]  Geoffrey E. Hinton.  Connectionist Learning Procedures , 1989, Artif. Intell..

[32]  Peter Seitz,et al.  Minimum class entropy: A maximum information approach to layered networks , 1989, Neural Networks.

[33]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[34]  Peter M. Todd,et al.  Designing Neural Networks using Genetic Algorithms , 1989, ICGA.

[35]  David H. Sharp,et al.  Scaling, machine learning, and genetic neural nets , 1989 .

[36]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[37]  Hiroaki Kitano,et al.  Designing Neural Networks Using Genetic Algorithms with Graph Generation System , 1990, Complex Syst..

[38]  David G. Stork,et al.  Evolution and Learning in Neural Networks , 1990, NIPS.

[39]  Mike Rudnick,et al.  A bibliography of the intersection of genetic search and artificial neural networks , 1990 .

[40]  Lawrence. Davis,et al.  Handbook Of Genetic Algorithms , 1990 .

[41]  Frank P. Coyle,et al.  Aaai '90 , 1990, IEEE Expert.

[42]  Hiroaki Kitano,et al.  Empirical Studies on the Speed of Convergence of Neural Network Training Using Genetic Algorithms , 1990, AAAI.

[43]  Stewart W. Wilson.  Perceptron redux: emergence of structure , 1990 .

[44]  Richard K. Belew,et al.  Evolution, Learning, and Culture: Computational Metaphors for Adaptive Algorithms , 1990, Complex Syst..

[45]  Richard K. Belew,et al.  Evolving networks: using the genetic algorithm with connectionist learning , 1990 .

[46]  Larry J. Eshelman,et al.  Using genetic search to exploit the emergent behavior of neural networks , 1990 .

[47]  G. Mani,et al.  Learning by gradient descent in function space , 1990, 1990 IEEE International Conference on Systems, Man, and Cybernetics Conference Proceedings.

[48]  Marcus Frean,et al.  The Upstart Algorithm: A Method for Constructing and Training Feedforward Neural Networks , 1990, Neural Computation.

[49]  Geoffrey E. Hinton,et al.  A time-delay neural network architecture for isolated word recognition , 1990, Neural Networks.

[50]  Stefano Nolfi,et al.  Econets: Neural networks that learn in an environment , 1990 .

[51]  Thomas Bäck,et al.  A Survey of Evolution Strategies , 1991, ICGA.

[52]  Jan Paredis,et al.  The evolution of behavior: some experiments , 1991 .

[53]  David B. Fogel.  An information criterion for optimal neural network selection , 1991, IEEE Trans. Neural Networks.

[54]  Sio Carlos,et al.  Evolving a learning algorithm for the binary perceptron , 1991 .

[55]  William A. Phillips,et al.  A Biologically Supported Error-Correcting Learning Rule , 1991, Neural Computation.

[56]  David J. Chalmers,et al.  The Evolution of Learning: An Experiment in Genetic Connectionism , 1991 .

[57]  J. Utans,et al.  Selecting neural network architectures via the prediction risk: application to corporate bond rating prediction , 1991, Proceedings First International Conference on Artificial Intelligence Applications on Wall Street.

[58]  Kazuo Kyuma,et al.  Weight quantization in Boltzmann machines , 1991, Neural Networks.

[59]  Yoshua Bengio,et al.  Learning a synaptic learning rule , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[60]  Kazuo Kyuma,et al.  A Quantized Back Propagation Learning Rule and Its Application to Optical Neural Network , 1991 .

[61]  Xin Yao,et al.  Simulated annealing with extended neighbourhood , 1991, Int. J. Comput. Math..

[62]  Jocelyn Sietsma,et al.  Creating artificial neural networks that generalize , 1991, Neural Networks.

[63]  Xin Yao,et al.  Optimization by Genetic Annealing , 1991 .

[64]  Nicholas J. Radcliffe,et al.  Genetic neural networks on MIMD computers , 1992 .

[65]  Markus Höhfeld,et al.  Learning with limited numerical precision using the cascade-correlation algorithm , 1992, IEEE Trans. Neural Networks.

[66]  Stefan Bornholdt,et al.  General asymmetric neural networks and structure design by genetic algorithms: a learning rule for temporal patterns , 1992, Proceedings of IEEE Systems Man and Cybernetics Conference - SMC.

[67]  AC Tose.  Cell , 1993, Cell.

[68]  Xin Yao,et al.  The Evolution of Connectionist Networks , 1994 .

[69]  Jeffrey L. Elman,et al.  Learning and Evolution in Neural Networks , 1994, Adapt. Behav..

[70]  D. Signorini,et al.  Neural networks , 1995, The Lancet.