ELeaRNT: Evolutionary Learning of Rich Neural Network Topologies

Abstract : In this paper we present ELeaRNT an evolutionary strategy which evolves rich neural network topologies in order to find an optimal domain specific non linear function approximator with a good generalization performance. The neural networks evolved by the algorithm have a feed forward topology with shortcut connections and arbitrary activation functions at each layer. This kind of topologies has not been thoroughly investigated in literature, but is particularly well suited for non linear regression tasks. The experimental results prove that, in such tasks, our algorithm can build, in a completely automated way, neural network topologies able to outperform classic neural network models designed by hand. Also when applied to classification problems, the performance of the obtained neural networks is fully comparable to that of classic neural networks and in some cases noticeably better.

[1]  Xin YaoComputational.  A Population-Based Learning Algorithm Which Learns BothArchitectures and Weights of Neural Networks , 1996 .

[2]  M. Stone.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[3]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[4]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[5]  Lars Kai Hansen,et al.  Neural Network Ensembles , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  Astro Teller,et al.  Neural Programming and an Internal Reinforcement Policy , 1996 .

[7]  Marc Schoenauer,et al.  Genetic Lander: An Experiment in Accurate Neuro-Genetic Control , 1994, PPSN.

[8]  Gary William Flake,et al.  Nonmonotonic activation functions in multilayer perceptrons , 1993 .

[9]  Peter J. B. Hancock,et al.  Genetic algorithms and permutation problems: a comparison of recombination operators for neural net structure specification , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[10]  Brian A. Telfer,et al.  Energy functions for minimizing misclassification error with minimum-complexity networks , 1994, Neural Networks.

[11]  Frank Kursawe,et al.  Evolution Strategies for Vector Optimization , 1992 .

[12]  C. Lee Giles,et al.  What Size Neural Network Gives Optimal Generalization? Convergence Properties of Backpropagation , 1998 .

[13]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[14]  Vasant Honavar,et al.  Evolutionary Design of Neural Architectures -- A Preliminary Taxonomy and Guide to Literature , 1995 .

[15]  Peter J. Angeline,et al.  Genetic programming and emergent intelligence , 1994 .

[16]  O. Mangasarian,et al.  Multisurface method of pattern separation for medical diagnosis applied to breast cytology. , 1990, Proceedings of the National Academy of Sciences of the United States of America.

[17]  Hiroaki Kitano,et al.  Designing Neural Networks Using Genetic Algorithms with Graph Generation System , 1990, Complex Syst..

[18]  G. Mani,et al.  Learning by gradient descent in function space , 1990, 1990 IEEE International Conference on Systems, Man, and Cybernetics Conference Proceedings.

[19]  Xin Yao,et al.  A review of evolutionary artificial neural networks , 1993, Int. J. Intell. Syst..

[20]  J. D. Schaffer,et al.  Combinations of genetic algorithms and neural networks: a survey of the state of the art , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[21]  A. Tsoi,et al.  The Performance of the Neocognitron with Various S-Cell and C-CellTransfer , 1992 .

[22]  Lutz Prechelt,et al.  A Set of Neural Network Benchmark Problems and Benchmarking Rules , 1994 .

[23]  Babak Hassibi,et al.  Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , 1992, NIPS.

[24]  I C G Campbell,et al.  Constructive learning techniques for designing neural network systems , 1998 .

[25]  Frank Kursawe,et al.  A Variant of Evolution Strategies for Vector Optimization , 1990, PPSN.

[26]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[27]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.