Effects of Occam's Razor in Evolving Sigma-Pi Neural Nets

Several evolutionary algorithms make use of hierarchical representations of variable size rather than linear strings of fixed length. Variable complexity of the structures provides an additional representational power which may widen the application domain of evolutionary algorithms. The price for this is, however, that the search space is open-ended and solutions may grow to arbitrarily large size. In this paper we study the effects of structural complexity of the solutions on their generalization performance by analyzing the fitness landscape of sigma-pi neural networks. The analysis suggests that smaller networks achieve, on average, better generalization accuracy than larger ones, thus confirming the usefulness of Occam's razor. A simple method for implementing the Occam's razor principle is described and shown to be effective in improving the generalization accuracy without limiting their learning capacity.

[1]  Hiroaki Kitano,et al.  Designing Neural Networks Using Genetic Algorithms with Graph Generation System , 1990, Complex Syst..

[2]  Tariq Samad,et al.  Towards the Genetic Synthesisof Neural Networks , 1989, ICGA.

[3]  L. Darrell Whitley,et al.  Genetic algorithms and neural networks: optimizing connections and connectivity , 1990, Parallel Comput..

[4]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[5]  D. E. Goldberg,et al.  Genetic Algorithms in Search, Optimization & Machine Learning , 1989 .

[6]  Byoung-Tak Zhang,et al.  Evolving Optimal Neural Networks Using Genetic Algorithms with Occam's Razor , 1993, Complex Syst..

[7]  Hillol Kargupta,et al.  System Identification with Evolving Polynomial Networks , 1991, ICGA.

[8]  Colin Giles,et al.  Learning, invariance, and generalization in high-order neural networks. , 1987, Applied optics.

[9]  J. David Schaffer,et al.  Proceedings of the third international conference on Genetic algorithms , 1989 .

[10]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[11]  J. Rissanen.  Stochastic Complexity and Modeling , 1986 .

[12]  Yaser S. Abu-Mostafa,et al.  The Vapnik-Chervonenkis Dimension: Information versus Complexity in Learning , 1989, Neural Computation.

[13]  Byoung-Tak Zhang,et al.  Genetic Programming of Minimal Neural Nets Using Occam's Razor , 1993, ICGA.

[14]  Heinz Mühlenbein,et al.  Predictive Models for the Breeder Genetic Algorithm I. Continuous Parameter Optimization , 1993, Evolutionary Computation.

[15]  Thomas Bäck,et al.  An Overview of Evolutionary Algorithms for Parameter Optimization , 1993, Evolutionary Computation.

[16]  Kenneth E. Kinnear,et al.  Generality and Difficulty in Genetic Programming: Evolving a Sort , 1993, ICGA.

[17]  Frédéric Gruau,et al.  Genetic synthesis of Boolean neural networks with a cell rewriting developmental process , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[18]  Hitoshi Iba,et al.  System Identification using Structured Genetic Algorithms , 1993, ICGA.

[19]  Byoung-Tak Zhang,et al.  Synthesis of sigma-pi neural networks by the breeder genetic programming , 1994, Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence.

[20]  David E. Rumelhart,et al.  Product Units: A Computationally Powerful and Biologically Plausible Extension to Backpropagation Networks , 1989, Neural Computation.

[21]  Ingo Rechenberg,et al.  Evolutionsstrategie : Optimierung technischer Systeme nach Prinzipien der biologischen Evolution , 1973 .

[22]  Walter Alden Tackett,et al.  Genetic Programming for Feature Discovery and Image Discrimination , 1993, ICGA.