Bayesian Methods for Efficient Genetic Programming

A Bayesian framework for genetic programming (GP) is presented. This is motivated by the observation that genetic programming iteratively searches populations of fitter programs and thus the information gained in the previous generation can be used in the next generation. The Bayesian GP makes use of Bayes theorem to estimate the posterior distribution of programs from their prior distribution and likelihood for the fitness data observed. Offspring programs are then generated by sampling from the posterior distribution by genetic variation operators. We present two GP algorithms derived from the Bayesian GP framework. One is the genetic programming with the adaptive Occam's razor (AOR) designed to evolve parsimonious programs. The other is the genetic programming with incremental data inheritance (IDI) designed to accelerate evolution by active selection of fitness cases. A multiagent learning task is used to demonstrate the effectiveness of the presented methods. In a series of experiments, AOR reduced solution complexity by 20% and IDI doubled evolution speed, both without loss of solution accuracy.

[1]  P. Nordin,et al.  Explicitly defined introns and destructive crossover in genetic programming , 1996 .

[2]  Una-May O'Reilly,et al.  Investigating the Generality of Automatically Defined Functions , 1996 .

[3]  Terence Soule,et al.  Code growth in genetic programming , 1996 .

[4]  Byoung-Tak Zhang,et al.  Genetic programming with incremental data inheritance , 1999 .

[5]  Wolfgang Banzhaf,et al.  Genetic Programming: An Introduction , 1997 .

[6]  Peter Nordin,et al.  The Effect of Extensive Use of the Mutation Operator on Generalization in Genetic Programming Using Sparse Data Sets , 1996, PPSN.

[7]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[8]  Jeffrey C. Hall,et al.  Advances in Genetics , 1947 .

[9]  John R. Koza,et al.  Scalable learning in genetic programming using automatic function definition , 1994 .

[10]  Riccardo Poli,et al.  On the Search Properties of Different Crossover Operators in Genetic Programming , 2001 .

[11]  Lee Spector,et al.  Simultaneous evolution of programs and their control structures , 1996 .

[12]  Terence Soule,et al.  Effects of Code Growth and Parsimony Pressure on Populations in Genetic Programming , 1998, Evolutionary Computation.

[13]  Tobias Blickle,et al.  Evolving Compact Solutions in Genetic Programming: A Case Study , 1996, PPSN.

[14]  Heinz Mühlenbein,et al.  The Science of Breeding and Its Application to the Breeder Genetic Algorithm (BGA) , 1993, Evolutionary Computation.

[15]  Peter Nordin,et al.  On Some Emergent Properties of Variable Size Evolutionary Algorithms , 1997 .

[16]  Justinian P. Rosca,et al.  Discovery of subroutines in genetic programming , 1996 .

[17]  Michèle Sebag,et al.  Evolutionary identification of macro-mechanical models , 1996 .

[18]  William B. Langdon,et al.  Size Fair and Homologous Tree Crossovers for Tree Genetic Programming , 2000, Genetic Programming and Evolvable Machines.

[19]  Peter Nordin,et al.  Genetic programming - An Introduction: On the Automatic Evolution of Computer Programs and Its Applications , 1998 .

[20]  David Andre,et al.  Automatically defined features: the simultaneous evolution of 2-dimensional feature detectors and an algorithm for using them , 1994 .

[21]  Hitoshi Iba,et al.  Genetic programming using a minimum description length principle , 1994 .

[22]  Riccardo Poli,et al.  Fitness Causes Bloat: Mutation , 1997, EuroGP.

[23]  W. Daniel Hillis,et al.  Co-evolving parasites improve simulated evolution as an optimization procedure , 1990 .

[24]  Peter J. Angeline,et al.  Genetic programming and emergent intelligence , 1994 .

[25]  Byoung-Tak Zhang,et al.  Genetic Programming of Minimal Neural Nets Using Occam's Razor , 1993, ICGA.

[26]  Eric V. Siegel.  Competitively evolving decision trees against fixed training cases for natural language processing , 1994 .

[27]  Peter Ross,et al.  Dynamic Training Subset Selection for Supervised Learning in Genetic Programming , 1994, PPSN.

[28]  Thomas Bäck,et al.  Evolutionary algorithms in theory and practice - evolution strategies, evolutionary programming, genetic algorithms , 1996 .

[29]  S. Luke,et al.  A Comparison of Crossover and Mutation in Genetic Programming , 1997 .

[30]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[31]  J. Pollack,et al.  Coevolving High-Level Representations , 1993 .

[32]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[33]  William B. Langdon,et al.  Size fair and homologous tree crossovers , 1999 .

[34]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[35]  Peter J. Angeline,et al.  Explicitly Defined Introns and Destructive Crossover in Genetic Programming , 1996 .

[36]  Byoung-Tak Zhang,et al.  Balancing Accuracy and Parsimony in Genetic Programming , 1995, Evolutionary Computation.

[37]  Byoung-Tak Zhang,et al.  Accelerated Learning by Active Example Selection , 1994, Int. J. Neural Syst..

[38]  Annie S. Wu,et al.  Empirical Studies of the Genetic Algorithm with Noncoding Segments , 1995, Evolutionary Computation.

[39]  Byoung-Tak Zhang.  A Bayesian framework for evolutionary computation , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[40]  Thomas Bäck,et al.  Evolutionary algorithms , 1992, SIGB.

[41]  J. Rissanen.  Stochastic Complexity and Modeling , 1986 .

[42]  Thomas Bäck,et al.  Evolutionary Algorithms in Theory and Practice , 1996 .

[43]  Kenneth E. Kinnear,et al.  Generality and Difficulty in Genetic Programming: Evolving a Sort , 1993, ICGA.

[44]  ProgrammingJustinian P. RoscaComputer.  Analysis of Complexity Drift in Genetic , 1997 .

[45]  Byoung-Tak Zhang,et al.  Evolutionary Induction of Sparse Neural Trees , 1997, Evolutionary Computation.