Effective structure learning for EDA via L1-regularizedbayesian networks

The Bayesian optimization algorithm (BOA) uses Bayesian networks to explore the dependencies between decision variables of an optimization problem in pursuit of both faster speed of convergence and better solution quality. In this paper, a novel method that learns the structure of Bayesian networks for BOA is proposed. The proposed method, called L1BOA, uses L1-regularized regression to find the candidate parents of each variable, which leads to a sparse but nearly optimized network structure. The proposed method improves the efficiency of the structure learning in BOA due to the reduction and automated control of network complexity introduced with L1-regularized learning. Experimental studies on different types of benchmark problems are carried out, which show that L1BOA outperforms the standard BOA when no a-priori knowledge about the problem structure is available, and nearly achieves the best performance of BOA that applies explicit complexity controls.

[1]  David Maxwell Chickering,et al.  Learning Bayesian Networks: The Combination of Knowledge and Statistical Data , 1994, Machine Learning.

[2]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[3]  Dimitri P. Bertsekas,et al.  Nonlinear Programming 2 , 2005 .

[4]  Gregory F. Cooper,et al.  A Bayesian Method for the Induction of Probabilistic Networks from Data , 1992 .

[5]  Daphne Koller,et al.  Efficient Structure Learning of Markov Networks using L1-Regularization , 2006, NIPS.

[6]  David E. Goldberg,et al.  Hierarchical Bayesian Optimization Algorithm , 2006, Scalable Optimization via Probabilistic Modeling.

[7]  N. Meinshausen,et al.  High-dimensional graphs and variable selection with the Lasso , 2006, math/0608017.

[8]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[9]  Martin Pelikan,et al.  Hierarchical Bayesian optimization algorithm: toward a new generation of evolutionary algorithms , 2010, SICE 2003 Annual Conference (IEEE Cat. No.03TH8734).

[10]  Mark W. Schmidt,et al.  Learning Graphical Model Structure Using L1-Regularization Paths , 2007, AAAI.

[11]  Richard E. Neapolitan,et al.  Learning Bayesian networks , 2007, KDD '07.

[12]  D. Hosmer,et al.  Applied Logistic Regression , 1991 .

[13]  David Maxwell Chickering,et al.  Learning Bayesian Networks is , 1994 .

[14]  J. A. Lozano,et al.  Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation , 2001 .

[15]  Jonathan L. Shapiro,et al.  Model Complexity vs. Performance in the Bayesian Optimization Algorithm , 2006, PPSN.

[16]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[17]  David E. Goldberg,et al.  A Survey of Optimization by Building and Using Probabilistic Models , 2002, Comput. Optim. Appl..

[18]  Kalyanmoy Deb,et al.  Analyzing Deception in Trap Functions , 1992, FOGA.

[19]  A. Ng.  Feature selection, L1 vs. L2 regularization, and rotational invariance , 2004, Twenty-first international conference on Machine learning - ICML '04.

[20]  Honglak Lee,et al.  Efficient L1 Regularized Logistic Regression , 2006, AAAI.

[21]  D. Goldberg,et al.  BOA: the Bayesian optimization algorithm , 1999 .