G-Prop-III: Global Optimization of Multilayer Perceptrons using an Evolutionary Algorithm

This paper proposes a new version of a method that attempts to solve the problem of finding appropriate size, initial weights and learning parameters for a single hidden layer Multilayer Perceptron (MLP) by combining a genetic algorithm and backpropagation. The GA selects the initial weights and the learning rate of the network, includes BP training as a mutation operator., and changes the number of neurons in the hidden layer through the application of specific genetic operators. The application of the G-Prop-III algorithm to several real-world and benchmark problems shows that MLPs evolved using G-Prop-III are smaller and achieve a higher level of generalization than other perceptron training algorithms, such as QuickPropagation or RPROP, and other evolutive algorithms, such as G-LVQ. It also shows some improvement over previous versions of the algorithm.

[1]  Scott E. Fahlman,et al.  An empirical study of learning speed in back-propagation networks , 1988 .

[2]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[3]  L. Darrell Whitley,et al.  The GENITOR Algorithm and Selection Pressure: Why Rank-Based Allocation of Reproductive Trials is Best , 1989, ICGA.

[4]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[5]  S. Fahlman.  Fast-learning variations on back propagation: an empirical study. , 1989 .

[6]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[7]  Peter M. Todd,et al.  Designing Neural Networks using Genetic Algorithms , 1989, ICGA.

[8]  O. Mangasarian,et al.  Pattern Recognition Via Linear Programming: Theory and Application to Medical Diagnosis , 1989 .

[9]  C. Jutten,et al.  Gal: Networks That Grow When They Learn and Shrink When They Forget , 1991 .

[10]  Vassilios Petridis,et al.  A hybrid genetic algorithm for training neural networks , 1992 .

[11]  Anna Maria Fanelli,et al.  A Method of Pruning Layered Feed-Forward Neural Networks , 1993, IWANN.

[12]  David White,et al.  GANNet: A Genetic Algorithm for Optimizing Topology and Weights in Neural Network Design , 1993, IWANN.

[13]  Juan Julián Merelo Guervós,et al.  Optimization of a Competitive Learning Neural Network by Genetic Algorithms , 1993, IWANN.

[14]  Martin A. Riedmiller,et al.  A direct adaptive method for faster backpropagation learning: the RPROP algorithm , 1993, IEEE International Conference on Neural Networks.

[15]  Gregory J. Wolff,et al.  Optimal Brain Surgeon: Extensions and performance comparisons , 1993, NIPS 1993.

[16]  Werner Kinnebrock,et al.  Accelerating the standard backpropagation method using a genetic approach , 1994, Neurocomputing.

[17]  Lutz Prechelt,et al.  PROBEN 1 - a set of benchmarks and benchmarking rules for neural network training algorithms , 1994 .

[18]  Hean-Lee Poh,et al.  Analysis of Pruning in Backpropagation Networks for Artificial and Real Worls Mapping Problems , 1995, IWANN.

[19]  Vasant Honavar,et al.  Evolutionary Design of Neural Architectures -- A Preliminary Taxonomy and Guide to Literature , 1995 .

[20]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1996, Springer Berlin Heidelberg.

[21]  Martin P. DeSimio,et al.  MLP iterative construction algorithm , 1997, Neurocomputing.

[22]  Michael Georgiopoulos,et al.  Coupling weight elimination with genetic algorithms to reduce network size and preserve generalization , 1997, Neurocomputing.

[23]  Rajesh Parekh,et al.  Constructive Neural Network Learning Algorithms for Multi-Category Real-Valued Pattern Classification , 1997 .

[24]  Jie Zhang,et al.  A Sequential Learning Approach for Single Hidden Layer Neural Networks , 1998, Neural Networks.

[25]  Xin Yao,et al.  Towards designing artificial neural networks by evolution , 1998 .

[26]  Ivanoe De Falco,et al.  Evolutionary Neural Networks for Nonlinear Dynamics Modeling , 1998, PPSN.

[27]  J. Merelo.  Automatic Classiication of Biological Particles from Electron-microscopy Images Using Conventional and Genetic-algorithm Optimized Learning Vector Quantization , 1998 .

[28]  Lars Kai Hansen,et al.  Neural classifier construction using regularization, pruning and test error estimation , 1998, Neural Networks.

[29]  Ernesto Tarantino,et al.  Optimizing Neural Networks for Time Series Prediction , 1999 .

[30]  Juan Julián Merelo Guervós,et al.  SA-Prop: Optimization of Multilayer Perceptron Parameters Using Simulated Annealing , 1999, IWANN.

[31]  Pedro Ángel Castillo Valdivieso,et al.  G-Prop-II: global optimization of multilayer perceptrons using GAs , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[32]  Juan Julián Merelo Guervós,et al.  G-Prop: Global optimization of multilayer perceptrons using GAs , 2000, Neurocomputing.