Latent Variable Model for Estimation of Distribution Algorithm Based on a Probabilistic Context-Free Grammar

Estimation of distribution algorithms are evolutionary algorithms using probabilistic techniques instead of traditional genetic operators. Recently, the application of probabilistic techniques to program and function evolution has received increasing attention, and this approach promises to provide a strong alternative to the traditional genetic programming techniques. Although a probabilistic context-free grammar (PCFG) is a widely used model for probabilistic program evolution, a conventional PCFG is not suitable for estimating interactions among nodes because of the context freedom assumption. In this paper, we have proposed a new evolutionary algorithm named programming with annotated grammar estimation based on a PCFG with latent annotations, which allows this context freedom assumption to be weakened. By applying the proposed algorithm to several computational problems, it is demonstrated that our approach is markedly more effective at estimating building blocks than prior approaches.

[1]  Michèle Sebag,et al.  Avoiding the Bloat with Stochastic Grammar-Based Genetic Programming , 2001, Artificial Evolution.

[2]  Kalyanmoy Deb,et al.  RapidAccurate Optimization of Difficult Problems Using Fast Messy Genetic Algorithms , 1993, ICGA.

[3]  Rafal Salustowicz,et al.  Probabilistic Incremental Program Evolution , 1997, Evolutionary Computation.

[4]  Ivan Tanev,et al.  Incorporating Learning Probabilistic Context-Sensitive Grammar in Genetic Programming for Efficient Evolution and Adaptation of Snakebot , 2005, EuroGP.

[5]  Hussein A. Abbass,et al.  Grammar model-based program evolution , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[6]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[7]  Peter Nordin,et al.  A compiling genetic programming system that directly manipulates the machine-code , 1994 .

[8]  Hitoshi Iba,et al.  Estimation of Bayesian Network for Program Generation , 2006 .

[9]  Hussein A. Abbass,et al.  Program Evolution with Explicit Learning: a New Framework for Program Automatic Synthesis , 2003 .

[10]  Peter A. N. Bosman,et al.  Grammar Transformations in an EDA for Genetic Programming , 2004 .

[11]  Matthew J. Beal.  Variational algorithms for approximate Bayesian inference , 2003 .

[12]  Aurora Trinidad Ramirez Pozo,et al.  Bayesian Automatic Programming , 2005, EuroGP.

[13]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[14]  Jun'ichi Tsujii,et al.  Probabilistic CFG with Latent Annotations , 2005, ACL.

[15]  Peter A. Whigham,et al.  Grammatically-based Genetic Programming , 1995 .

[16]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[17]  Riccardo Poli,et al.  A Linear Estimation-of-Distribution GP System , 2008, EuroGP.

[18]  Hussein A. Abbass,et al.  AntTAG: a new method to compose computer programs using colonies of ants , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[19]  Kenichi Kurihara,et al.  An Application of the Variational Bayesian Approach to Probabilistic Context-Free Grammars , 2004 .

[20]  G. Harik.  Linkage Learning via Probabilistic Modeling in the ECGA , 1999 .

[21]  Hitoshi Iba,et al.  Probabilistic distribution models for EDA-based GP , 2005, GECCO '05.

[22]  D. Goldberg,et al.  Escaping hierarchical traps with competent genetic algorithms , 2001 .

[23]  Ivan Tanev,et al.  Implications of Incorporating Learning Probabilistic Context-sensitive Grammar in Genetic Programming on Evolvability of Adaptive Locomotion Gaits of Snakebot , 2004 .

[24]  Peter A. Whigham.  Inductive bias and genetic programming , 1995 .

[25]  Hussein A. Abbass,et al.  A Survey of Probabilistic Model Building Genetic Programming , 2006, Scalable Optimization via Probabilistic Modeling.

[26]  D. Goldberg,et al.  BOA: the Bayesian optimization algorithm , 1999 .

[27]  Andreas Stolcke,et al.  Bayesian learning of probabilistic language models , 1994 .

[28]  W. Langdon.  An Analysis of the MAX Problem in Genetic Programming , 1997 .

[29]  Pedro Larrañaga,et al.  Estimation of Distribution Algorithms , 2002, Genetic Algorithms and Evolutionary Computation.

[30]  P. Ross,et al.  An adverse interaction between crossover and restricted tree depth in genetic programming , 1996 .

[31]  Hitoshi Iba,et al.  A Bayesian Network Approach to Program Generation , 2008, IEEE Transactions on Evolutionary Computation.

[32]  Melanie Mitchell,et al.  The royal road for genetic algorithms: Fitness landscapes and GA performance , 1991 .

[33]  Ben Goertzel,et al.  Learning computer programs with the bayesian optimization algorithm , 2005, GECCO '05.

[34]  Moshe Looks,et al.  Scalable estimation-of-distribution program evolution , 2007, GECCO '07.

[35]  J. A. Lozano,et al.  Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation , 2001 .

[36]  Shumeet Baluja,et al.  A Method for Integrating Genetic Search Based Function Optimization and Competitive Learning , 1994 .

[37]  H. Iba,et al.  Estimation of distribution programming based on Bayesian network , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[38]  Mark Wineberg,et al.  A Representation Scheme To Perform Program Induction in a Canonical Genetic Algorithm , 1994, PPSN.

[39]  William F. Punch.  HOW EFFECTIVE ARE MULTIPLE POPULATIONS IN GENETIC PROGRAMMING , 1998 .

[40]  D. Goldberg,et al.  Probabilistic Model Building and Competent Genetic Programming , 2003 .

[41]  Hagai Attias,et al.  Inferring Parameters and Structure of Latent Variable Models by Variational Bayes , 1999, UAI.