Probabilistic Contextual and Structural Dependencies Learning in Grammar-Based Genetic Programming

Abstract Genetic Programming is a method to automatically create computer programs based on the principles of evolution. The problem of deceptiveness caused by complex dependencies among components of programs is challenging. It is important because it can misguide Genetic Programming to create suboptimal programs. Besides, a minor modification in the programs may lead to a notable change in the program behaviours and affect the final outputs. This article presents Grammar-Based Genetic Programming with Bayesian Classifiers (GBGPBC) in which the probabilistic dependencies among components of programs are captured using a set of Bayesian network classifiers. Our system was evaluated using a set of benchmark problems (the deceptive maximum problems, the royal tree problems, and the bipolar asymmetric royal tree problems). It was shown to be often more robust and more efficient in searching the best programs than other related Genetic Programming approaches in terms of the total number of fitness evaluation. We studied what factors affect the performance of GBGPBC and discovered that robust variants of GBGPBC were consistently weakly correlated with some complexity measures. Furthermore, our approach has been applied to learn a ranking program on a set of customers in direct marketing. Our suggested solutions help companies to earn significantly more when compared with other solutions produced by several well-known machine learning algorithms, such as neural networks, logistic regression, and Bayesian networks.

[1]  Michael O'Neill,et al.  Grammatical Evolution: Evolving Programs for an Arbitrary Language , 1998, EuroGP.

[2]  Shingo Mabu,et al.  Use of infeasible individuals in probabilistic model building genetic network programming , 2011, GECCO '11.

[3]  Kwong-Sak Leung,et al.  Probabilistic grammar-based neuroevolution for physiological signal classification of ventricular tachycardia , 2019, Expert Syst. Appl..

[4]  P. Ross,et al.  An adverse interaction between crossover and restricted tree depth in genetic programming , 1996 .

[5]  Barbara Webb,et al.  Swarm Intelligence: From Natural to Artificial Systems , 2002, Connect. Sci..

[6]  Ken Kennedy,et al.  Optimizing Compilers for Modern Architectures: A Dependence-based Approach , 2001 .

[7]  Hitoshi Iba,et al.  A Bayesian Network Approach to Program Generation , 2008, IEEE Transactions on Evolutionary Computation.

[8]  Rafal Salustowicz,et al.  Probabilistic Incremental Program Evolution , 1997, Evolutionary Computation.

[9]  Hussein A. Abbass,et al.  Program evolution with explicit learning , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[10]  M. O'Neill,et al.  Grammatical evolution , 2001, GECCO '09.

[11]  Yoshihiko Hasegawa.  Programming with Annotated Grammar Estimation , 2012 .

[12]  William F. Punch.  HOW EFFECTIVE ARE MULTIPLE POPULATIONS IN GENETIC PROGRAMMING , 1998 .

[13]  Dan Boneh,et al.  On genetic algorithms , 1995, COLT '95.

[14]  John R. Koza,et al.  Automated synthesis of analog electrical circuits by means of genetic programming , 1997, IEEE Trans. Evol. Comput..

[15]  Maarten Keijzer,et al.  Grammatical Evolution Rules: The Mod and the Bucket Rule , 2002, EuroGP.

[16]  Peter A. Whigham,et al.  Grammar-based Genetic Programming: a survey , 2010, Genetic Programming and Evolvable Machines.

[17]  Taylor L. Booth,et al.  Applying Probability Measures to Abstract Languages , 1973, IEEE Transactions on Computers.

[18]  Heinz Mühlenbein,et al.  FDA -A Scalable Evolutionary Algorithm for the Optimization of Additively Decomposed Functions , 1999, Evolutionary Computation.

[19]  Hitoshi Iba,et al.  Binary encoding for prototype tree of probabilistic model building GP , 2009, GECCO.

[20]  Michèle Sebag,et al.  Avoiding the Bloat with Stochastic Grammar-Based Genetic Programming , 2001, Artificial Evolution.

[21]  H. Iba,et al.  Estimation of distribution programming based on Bayesian network , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[22]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[23]  Ray J. Solomonoff,et al.  Complexity-based induction systems: Comparisons and convergence theorems , 1978, IEEE Trans. Inf. Theory.

[24]  Hussein A. Abbass,et al.  Grammar model-based program evolution , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[25]  Hagai Attias,et al.  Inferring Parameters and Structure of Latent Variable Models by Variational Bayes , 1999, UAI.

[26]  Aurora Trinidad Ramirez Pozo,et al.  Bayesian Automatic Programming , 2005, EuroGP.

[27]  Peter A. Whigham.  Inductive bias and genetic programming , 1995 .

[28]  Peter A. N. Bosman,et al.  Grammar Transformations in an EDA for Genetic Programming , 2004 .

[29]  Peter A. Whigham,et al.  Grammatically-based Genetic Programming , 1995 .

[30]  Kwong-Sak Leung,et al.  Grammar-Based Genetic Programming with Bayesian network , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[31]  Danushka Bollegala,et al.  Probabilistic model building GP with Belief propagation , 2012, 2012 IEEE Congress on Evolutionary Computation.

[32]  Hitoshi Iba,et al.  Estimation of Bayesian Network for Program Generation , 2006 .

[33]  Franz Rothlauf,et al.  On the Locality of Standard Search Operators in Grammatical Evolution , 2014, PPSN.

[34]  D. E. Goldberg,et al.  Simple Genetic Algorithms and the Minimal, Deceptive Problem , 1987 .

[35]  Ben Goertzel,et al.  Learning computer programs with the bayesian optimization algorithm , 2005, GECCO '05.

[36]  Nuno Lourenço,et al.  Unveiling the properties of structured grammatical evolution , 2016, Genetic Programming and Evolvable Machines.

[37]  Martin Pelikan,et al.  Hierarchical Bayesian optimization algorithm: toward a new generation of evolutionary algorithms , 2010, SICE 2003 Annual Conference (IEEE Cat. No.03TH8734).

[38]  Prabhas Chongstitvatana,et al.  Multi-objective Combinatorial Optimisation with Coincidence algorithm , 2009, 2009 IEEE Congress on Evolutionary Computation.

[39]  Anthony Brabazon,et al.  Grammatical Differential Evolution , 2006, IC-AI.

[40]  Kwong-Sak Leung,et al.  Data Mining Using Grammar Based Genetic Programming and Applications , 2000 .

[41]  Hitoshi Iba,et al.  Latent Variable Model for Estimation of Distribution Algorithm Based on a Probabilistic Context-Free Grammar , 2009, IEEE Transactions on Evolutionary Computation.

[42]  Kwong-Sak Leung,et al.  Grammar-based genetic programming with dependence learning and bayesian network classifier , 2014, GECCO.

[43]  Paul A. Viola,et al.  MIMIC: Finding Optima by Estimating Probability Densities , 1996, NIPS.

[44]  Franz Rothlauf,et al.  On the Locality of Grammatical Evolution , 2006, EuroGP.

[45]  Claude E. Shannon,et al.  A mathematical theory of communication , 1948, MOCO.

[46]  Siddhartha Bhattacharyya,et al.  Direct Marketing Performance Modeling Using Genetic Algorithms , 1999, INFORMS J. Comput..

[47]  Nguyen Xuan Hoai,et al.  Probabilistic model building in genetic programming: a critical review , 2013, Genetic Programming and Evolvable Machines.

[48]  Franz Rothlauf,et al.  DAE-GP: denoising autoencoder LSTM networks as probabilistic models in estimation of distribution genetic programming , 2020, GECCO.

[49]  David Heckerman,et al.  A Tutorial on Learning with Bayesian Networks , 1999, Innovations in Bayesian Networks.

[50]  Kwong-Sak Leung,et al.  Applying logic grammars to induce sub-functions in genetic programming , 1995, Proceedings of 1995 IEEE International Conference on Evolutionary Computation.

[51]  Lawrence. Davis,et al.  Handbook Of Genetic Algorithms , 1990 .

[52]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[53]  J. A. Lozano,et al.  Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation , 2001 .

[54]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[55]  D. Goldberg,et al.  Probabilistic Model Building and Competent Genetic Programming , 2003 .

[56]  Corso Elvezia.  Probabilistic Incremental Program Evolution , 1997 .

[57]  Nir Friedman,et al.  Bayesian Network Classifiers , 1997, Machine Learning.

[58]  Anthony Brabazon,et al.  GEVA: grammatical evolution in Java , 2008, SEVO.

[59]  Michael O'Neill,et al.  Grammatical evolution - evolutionary automatic programming in an arbitrary language , 2003, Genetic programming.

[60]  Shumeet Baluja,et al.  A Method for Integrating Genetic Search Based Function Optimization and Competitive Learning , 1994 .

[61]  Zbigniew Michalewicz,et al.  Evolutionary Algorithms for Constrained Parameter Optimization Problems , 1996, Evolutionary Computation.

[62]  J. Powell,et al.  c ○ 2004 The Review of Economic Studies Limited Endogeneity in Semiparametric Binary Response Models , 2001 .