Probabilistic Contextual and Structural Dependencies Learning in Grammar-Based Genetic Programming

Genetic Programming is a method to automatically create computer programs based on the principles of evolution. The problem of deceptiveness caused by complex dependencies among components of programs is challenging. It is important because it can misguide Genetic Programming to create sub-optimal programs. Besides, a minor modification in the programs may lead to a notable change in the program behaviours and affect the final outputs. This paper presents Grammar-based Genetic Programming with Bayesian Classifiers (GBGPBC) in which the probabilistic dependencies among components of programs are captured using a set of Bayesian network classifiers. Our system was evaluated using a set of benchmark problems (the deceptive maximum problems, the royal tree problems, and the bipolar asymmetric royal tree problems). It was shown to be often more robust and more efficient in searching the best programs than other related Genetic Programming approaches in terms of the total number of fitness evaluation. We studied what factors affect the performance of GBGPBC and discovered that robust variants of GBGPBC were consistently weakly correlated with some complexity measures. Furthermore, our approach has been applied to learn a ranking program on a set of customers in direct marketing. Our suggested solutions help companies to earn significantly more when compared with other solutions produced by several well-known machine learning algorithms, such as neural networks, logistic regression, and Bayesian networks.

[1]  Shingo Mabu,et al.  Use of infeasible individuals in probabilistic model building genetic network programming , 2011, GECCO '11.

[2]  Paul A. Viola,et al.  MIMIC: Finding Optima by Estimating Probability Densities , 1996, NIPS.

[3]  J. A. Lozano,et al.  Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation , 2001 .

[4]  Hussein A. Abbass,et al.  Program evolution with explicit learning , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[5]  Yoshihiko Hasegawa.  Programming with Annotated Grammar Estimation , 2012 .

[6]  Shumeet Baluja,et al.  A Method for Integrating Genetic Search Based Function Optimization and Competitive Learning , 1994 .

[7]  Franz Rothlauf,et al.  DAE-GP: denoising autoencoder LSTM networks as probabilistic models in estimation of distribution genetic programming , 2020, GECCO.

[8]  Hitoshi Iba,et al.  Latent Variable Model for Estimation of Distribution Algorithm Based on a Probabilistic Context-Free Grammar , 2009, IEEE Transactions on Evolutionary Computation.

[9]  J. Powell,et al.  c ○ 2004 The Review of Economic Studies Limited Endogeneity in Semiparametric Binary Response Models , 2001 .

[10]  Michèle Sebag,et al.  Avoiding the Bloat with Stochastic Grammar-Based Genetic Programming , 2001, Artificial Evolution.

[11]  Danushka Bollegala,et al.  Probabilistic model building GP with Belief propagation , 2012, 2012 IEEE Congress on Evolutionary Computation.

[12]  Kwong-Sak Leung,et al.  Data Mining Using Grammar Based Genetic Programming and Applications , 2000 .

[13]  John R. Koza,et al.  Automated synthesis of analog electrical circuits by means of genetic programming , 1997, IEEE Trans. Evol. Comput..

[14]  Lawrence. Davis,et al.  Handbook Of Genetic Algorithms , 1990 .

[15]  Barbara Webb,et al.  Swarm Intelligence: From Natural to Artificial Systems , 2002, Connect. Sci..

[16]  Corso Elvezia.  Probabilistic Incremental Program Evolution , 1997 .

[17]  Hitoshi Iba,et al.  A Bayesian Network Approach to Program Generation , 2008, IEEE Transactions on Evolutionary Computation.

[18]  Aurora Trinidad Ramirez Pozo,et al.  Bayesian Automatic Programming , 2005, EuroGP.

[19]  Kwong-Sak Leung,et al.  Grammar-based genetic programming with dependence learning and bayesian network classifier , 2014, GECCO.

[20]  P. Ross,et al.  An adverse interaction between crossover and restricted tree depth in genetic programming , 1996 .

[21]  Anthony Brabazon,et al.  GEVA: grammatical evolution in Java , 2008, SEVO.

[22]  H. Iba,et al.  Estimation of distribution programming based on Bayesian network , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[23]  Kwong-Sak Leung,et al.  Probabilistic grammar-based neuroevolution for physiological signal classification of ventricular tachycardia , 2019, Expert Syst. Appl..

[24]  Peter A. N. Bosman,et al.  Grammar Transformations in an EDA for Genetic Programming , 2004 .

[25]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[26]  Michael O'Neill,et al.  Grammatical evolution - evolutionary automatic programming in an arbitrary language , 2003, Genetic programming.

[27]  Hussein A. Abbass,et al.  Grammar model-based program evolution , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[28]  Dan Boneh,et al.  On genetic algorithms , 1995, COLT '95.

[29]  Kwong-Sak Leung,et al.  Grammar-Based Genetic Programming with Bayesian network , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[30]  Hitoshi Iba,et al.  Estimation of Bayesian Network for Program Generation , 2006 .

[31]  William F. Punch.  HOW EFFECTIVE ARE MULTIPLE POPULATIONS IN GENETIC PROGRAMMING , 1998 .

[32]  D. Goldberg,et al.  Probabilistic Model Building and Competent Genetic Programming , 2003 .

[33]  Peter A. Whigham,et al.  Grammar-based Genetic Programming: a survey , 2010, Genetic Programming and Evolvable Machines.

[34]  Peter A. Whigham.  Inductive bias and genetic programming , 1995 .

[35]  Ken Kennedy,et al.  Optimizing Compilers for Modern Architectures: A Dependence-based Approach , 2001 .

[36]  Anthony Brabazon,et al.  Grammatical Differential Evolution , 2006, IC-AI.

[37]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[38]  Maarten Keijzer,et al.  Grammatical Evolution Rules: The Mod and the Bucket Rule , 2002, EuroGP.

[39]  Peter A. Whigham,et al.  Grammatically-based Genetic Programming , 1995 .

[40]  Kwong-Sak Leung,et al.  Applying logic grammars to induce sub-functions in genetic programming , 1995, Proceedings of 1995 IEEE International Conference on Evolutionary Computation.

[41]  Franz Rothlauf,et al.  On the Locality of Grammatical Evolution , 2006, EuroGP.

[42]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[43]  Hitoshi Iba,et al.  Binary encoding for prototype tree of probabilistic model building GP , 2009, GECCO.

[44]  Zbigniew Michalewicz,et al.  Evolutionary Algorithms for Constrained Parameter Optimization Problems , 1996, Evolutionary Computation.

[45]  Franz Rothlauf,et al.  On the Locality of Standard Search Operators in Grammatical Evolution , 2014, PPSN.

[46]  Martin Pelikan,et al.  Hierarchical Bayesian optimization algorithm: toward a new generation of evolutionary algorithms , 2010, SICE 2003 Annual Conference (IEEE Cat. No.03TH8734).

[47]  Nguyen Xuan Hoai,et al.  Probabilistic model building in genetic programming: a critical review , 2013, Genetic Programming and Evolvable Machines.

[48]  Siddhartha Bhattacharyya,et al.  Direct Marketing Performance Modeling Using Genetic Algorithms , 1999, INFORMS J. Comput..

[49]  Nuno Lourenço,et al.  Unveiling the properties of structured grammatical evolution , 2016, Genetic Programming and Evolvable Machines.

[50]  Prabhas Chongstitvatana,et al.  Multi-objective Combinatorial Optimisation with Coincidence algorithm , 2009, 2009 IEEE Congress on Evolutionary Computation.

[51]  Nir Friedman,et al.  Bayesian Network Classifiers , 1997, Machine Learning.

[52]  Michael O'Neill,et al.  Grammatical Evolution: Evolving Programs for an Arbitrary Language , 1998, EuroGP.

[53]  Heinz Mühlenbein,et al.  FDA -A Scalable Evolutionary Algorithm for the Optimization of Additively Decomposed Functions , 1999, Evolutionary Computation.

[54]  D. E. Goldberg,et al.  Simple Genetic Algorithms and the Minimal, Deceptive Problem , 1987 .

[55]  Hagai Attias,et al.  Inferring Parameters and Structure of Latent Variable Models by Variational Bayes , 1999, UAI.

[56]  Taylor L. Booth,et al.  Applying Probability Measures to Abstract Languages , 1973, IEEE Transactions on Computers.

[57]  Ben Goertzel,et al.  Learning computer programs with the bayesian optimization algorithm , 2005, GECCO '05.

[58]  Ray J. Solomonoff,et al.  Complexity-based induction systems: Comparisons and convergence theorems , 1978, IEEE Trans. Inf. Theory.