Learning Bayesian Networks with Non-Decomposable Scores

Modern approaches for optimally learning Bayesian network structures require decomposable scores. Such approaches include those based on dynamic programming and heuristic search methods. These approaches operate in a search space called the order graph, which has been investigated extensively in recent years. In this paper, we break from this tradition, and show that one can effectively learn structures using non-decomposable scores by exploring a more complex search space that leverages state-of-the-art learning systems based on order graphs. We show how the new search space can be used to learn with priors that are not structure-modular (a particular class of non-decomposable scores). We also show that it can be used to efficiently enumerate the \(k\)-best structures, in time that can be up to three orders of magnitude faster, compared to existing approaches.

[1]  Rina Dechter,et al.  Search Algorithms for m Best Solutions for Graphical Models , 2012, AAAI.

[2]  David Maxwell Chickering,et al.  Learning Bayesian Networks: The Combination of Knowledge and Statistical Data , 1994, Machine Learning.

[3]  Nils J. Nilsson,et al.  A Formal Basis for the Heuristic Determination of Minimum Cost Paths , 1968, IEEE Trans. Syst. Sci. Cybern..

[4]  Nir Friedman,et al.  Being Bayesian about Network Structure , 2000, UAI.

[5]  Nathan R. Sturtevant,et al.  Partial-Expansion A* with Selective Node Generation , 2012, SOCS.

[6]  Andrew W. Moore,et al.  Finding optimal Bayesian networks by dynamic programming , 2005 .

[7]  Gregory F. Cooper,et al.  A Bayesian method for the induction of probabilistic networks from data , 1992, Machine Learning.

[8]  James Cussens,et al.  Bayesian network learning with cutting planes , 2011, UAI.

[9]  Changhe Yuan,et al.  Learning Optimal Bayesian Networks: A Shortest Path Perspective , 2013, J. Artif. Intell. Res..

[10]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[11]  Wray L. Buntine.  Theory Refinement on Bayesian Networks , 1991, UAI.

[12]  Jim Q. Smith,et al.  Exact estimation of multiple directed acyclic graphs , 2014, Stat. Comput..

[13]  Wai Lam,et al.  LEARNING BAYESIAN BELIEF NETWORKS: AN APPROACH BASED ON THE MDL PRINCIPLE , 1994, Comput. Intell..

[14]  David Heckerman,et al.  Learning Bayesian Networks: Search Methods and Experimental Results , 1995 .

[15]  Tomi Silander,et al.  A Simple Approach for Finding the Globally Optimal Bayesian Network Structure , 2006, UAI.

[16]  Joe Suzuki,et al.  A Construction of Bayesian Networks from Databases Based on an MDL Principle , 1993, UAI.

[17]  Teruhisa Miura,et al.  A* with Partial Expansion for Large Branching Factor Problems , 2000, AAAI/IAAI.

[18]  Mikko Koivisto,et al.  Annealed Importance Sampling for Structure Learning in Bayesian Networks , 2013, IJCAI.

[19]  Jin Tian,et al.  Bayesian model averaging using the k -best Bayesian network structures , 2010, UAI 2010.

[20]  Mikko Koivisto,et al.  Exact Bayesian Structure Discovery in Bayesian Networks , 2004, J. Mach. Learn. Res..

[21]  G. Brightwell,et al.  Counting linear extensions , 1991 .

[22]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[23]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[24]  Daphne Koller,et al.  Ordering-Based Search: A Simple and Effective Algorithm for Learning Bayesian Networks , 2005, UAI.

[25]  James Cussens,et al.  An upper bound for BDeu local scores , 2012 .

[26]  Remco R. Bouckaert,et al.  Probalistic Network Construction Using the Minimum Description Length Principle , 1993, ECSQARU.

[27]  Changhe Yuan,et al.  Memory-Efficient Dynamic Programming for Learning Optimal Bayesian Networks , 2011, AAAI.

[28]  Changhe Yuan,et al.  Learning Optimal Bayesian Networks Using A* Search , 2011, IJCAI.

[29]  Brandon M. Malone,et al.  Predicting the Hardness of Learning Bayesian Networks , 2014, AAAI.

[30]  Jin Tian,et al.  A Branch-and-Bound Algorithm for MDL Learning Bayesian Networks , 2000, UAI.

[31]  Qiang Ji,et al.  Efficient Structure Learning of Bayesian Networks using Constraints , 2011, J. Mach. Learn. Res..

[32]  Adnan Darwiche,et al.  Modeling and Reasoning with Bayesian Networks , 2009 .

[33]  Tommi S. Jaakkola,et al.  Learning Bayesian Network Structure using LP Relaxations , 2010, AISTATS.

[34]  James Cussens,et al.  Maximum Likelihood Pedigree Reconstruction Using Integer Linear Programming , 2013, Genetic epidemiology.