Searching for Bayesian Network Structures in the Space of Restricted Acyclic Partially Directed Graphs

Although many algorithms have been designed to construct Bayesian network structures using different approaches and principles, they all employ only two methods: those based on independence criteria, and those based on a scoring function and a search procedure (although some methods combine the two). Within the score+search paradigm, the dominant approach uses local search methods in the space of directed acyclic graphs (DAGs), where the usual choices for defining the elementary modifications (local changes) that can be applied are arc addition, arc deletion, and arc reversal. In this paper, we propose a new local search method that uses a different search space, and which takes account of the concept of equivalence between network structures: restricted acyclic partially directed graphs (RPDAGs). In this way, the number of different configurations of the search space is reduced, thus improving efficiency. Moreover, although the final result must necessarily be a local optimum given the nature of the search method, the topology of the new search space, which avoids making early decisions about the directions of the arcs, may help to find better local optima than those obtained by searching in the DAG space. Detailed results of the evaluation of the proposed search method on several test problems, including the well-known Alarm Monitoring System, are also presented.

[1]  Luis M. de Campos,et al.  On the use of independence relationships for learning simplified belief networks , 1997, Int. J. Intell. Syst..

[2]  Jin Tian,et al.  A Branch-and-Bound Algorithm for MDL Learning Bayesian Networks , 2000, UAI.

[3]  Jie Cheng,et al.  Learning Bayesian Networks from Data: An Efficient Approach Based on Information Theory , 1999 .

[4]  José Miguel Puerta Callejón.  Métodos locales y distribuidos para la construcción de redes de creencia estáticas y dinámicas , 2001 .

[5]  A. H. Murphy,et al.  Hailfinder: A Bayesian system for forecasting severe weather , 1996 .

[6]  David Maxwell Chickering,et al.  Learning Bayesian Networks: The Combination of Knowledge and Statistical Data , 1994, Machine Learning.

[7]  Robert Castelo,et al.  Improved learning of Bayesian networks , 2001, UAI.

[8]  C. N. Liu,et al.  Approximating discrete probability distributions with dependence trees , 1968, IEEE Trans. Inf. Theory.

[9]  David Maxwell Chickering,et al.  Learning Equivalence Classes of Bayesian Network Structures , 1996, UAI.

[10]  Jose Miguel Puerta,et al.  Local Search Methods for Learning Bayesian Networks Using a Modified Neighborhood in the Space of DAGs , 2002, IBERAMIA.

[11]  Luis M. de Campos,et al.  A hybrid methodology for learning belief networks: BENEDICT , 2001, Int. J. Approx. Reason..

[12]  J. Huete,et al.  On the use of independence relationships for learning simplified belief networks , 1997 .

[13]  Luis M. de Campos,et al.  A new approach for learning belief networks using independence criteria , 2000, Int. J. Approx. Reason..

[14]  D. Madigan,et al.  A characterization of Markov equivalence classes for acyclic digraphs , 1997 .

[15]  Tom Burr,et al.  Causation, Prediction, and Search , 2003, Technometrics.

[16]  Nir Friedman,et al.  Being Bayesian about Network Structure , 2000, UAI.

[17]  Solomon Kullback,et al.  Information Theory and Statistics , 1970, The Mathematical Gazette.

[18]  Gregory F. Cooper,et al.  A Bayesian method for the induction of probabilistic networks from data , 1992, Machine Learning.

[19]  Judea Pearl,et al.  Equivalence and Synthesis of Causal Models , 1990, UAI.

[20]  Christopher Meek,et al.  Learning Bayesian Networks with Discrete Variables from Data , 1995, KDD.

[21]  David Maxwell Chickering,et al.  A Transformational Characterization of Equivalent Bayesian Network Structures , 1995, UAI.

[22]  Luis M. de Campos,et al.  Learning bayesian networks by ant colony optimisation: searching in two different spaces , 2002 .

[23]  Paul Munteanu,et al.  Efficient Score-Based Learning of Equivalence Classes of Bayesian Networks , 2000, PKDD.

[24]  R. Bouckaert.  Bayesian belief networks : from construction to inference , 1995 .

[25]  Pedro Larrañaga,et al.  Learning Bayesian network structures by searching for the best ordering with genetic algorithms , 1996, IEEE Trans. Syst. Man Cybern. Part A.

[26]  Paola Sebastiani,et al.  Learning Bayesian Networks from Incomplete Databases , 1997, UAI.

[27]  D. Madigan,et al.  Bayesian model averaging and model selection for markov equivalence classes of acyclic digraphs , 1996 .

[28]  Wray L. Buntine.  Operations for Learning with Graphical Models , 1994, J. Artif. Intell. Res..

[29]  Judea Pearl,et al.  The recovery of causal poly-trees from statistical data , 1987, Int. J. Approx. Reason..

[30]  Wai Lam,et al.  LEARNING BAYESIAN BELIEF NETWORKS: AN APPROACH BASED ON THE MDL PRINCIPLE , 1994, Comput. Intell..

[31]  Marek J. Druzdzel,et al.  A Hybrid Anytime Algorithm for the Construction of Causal Models From Sparse Data , 1999, UAI.

[32]  David Heckerman,et al.  Bayesian Networks for Knowledge Discovery , 1996, Advances in Knowledge Discovery and Data Mining.

[33]  M. Tarsi,et al.  A simple algorithm to construct a consistent extension of a partially oriented graph , 1992 .

[34]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[35]  Pedro Larrañaga,et al.  Learning Bayesian networks in the space of structures by estimation of distribution algorithms , 2003, Int. J. Intell. Syst..

[36]  Stuart J. Russell,et al.  Adaptive Probabilistic Networks with Hidden Variables , 1997, Machine Learning.

[37]  Jie Cheng,et al.  An Algorithm for Bayesian Belief Network Construction from Data , 2004 .

[38]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[39]  Luis M. de Campos,et al.  Learning Right Sized Belief Networks by Means of a Hybrid Methodology , 2000, PKDD.

[40]  N. Wermuth,et al.  Graphical and recursive models for contingency tables , 1983 .

[41]  Joe Suzuki,et al.  A Construction of Bayesian Networks from Databases Based on an MDL Principle , 1993, UAI.

[42]  Luis M. de Campos,et al.  Independency relationships and learning algorithms for singly connected networks , 1998, J. Exp. Theor. Artif. Intell..

[43]  Gregory F. Cooper,et al.  The ALARM Monitoring System: A Case Study with two Probabilistic Inference Techniques for Belief Networks , 1989, AIME.

[44]  Ron Kohavi,et al.  Data Mining Using MLC a Machine Learning Library in C++ , 1996, Int. J. Artif. Intell. Tools.

[45]  Jose Miguel Puerta,et al.  Stochastic Local Algorithms for Learning Belief Networks: Searching in the Space of the Orderings , 2001, ECSQARU.

[46]  Edward H. Herskovits,et al.  Computer-based probabilistic-network construction , 1992 .

[47]  Michael D. Perlman,et al.  Enumerating Markov Equivalence Classes of Acyclic Digraph Models , 2001, UAI.

[48]  Nir Friedman,et al.  Learning Bayesian Network Structure from Massive Datasets: The "Sparse Candidate" Algorithm , 1999, UAI.

[49]  Joe Suzuki,et al.  Learning Bayesian Belief Networks Based on the Minimum Description Length Principle: An Efficient Algorithm Using the B & B Technique , 1996, ICML.

[50]  J. Pearl,et al.  Learning simple causal structures , 1993 .

[51]  Ron Kohavi,et al.  MLC++: a machine learning library in C++ , 1994, Proceedings Sixth International Conference on Tools with Artificial Intelligence. TAI 94.

[52]  Harald Steck.  On the Use of Skeletons when Learning in Bayesian Networks , 2000, UAI.

[53]  Moninder Singh,et al.  Construction of Bayesian network structures from data: A brief survey and an efficient algorithm , 1995, Int. J. Approx. Reason..

[54]  Judea Pearl,et al.  Causal networks: semantics and expressiveness , 2013, UAI.

[55]  David Heckerman,et al.  Learning Bayesian Networks: Search Methods and Experimental Results , 1995 .

[56]  David Heckerman,et al.  A Characterization of the Dirichlet Distribution with Application to Learning Bayesian Networks , 1995, UAI.

[57]  Kwong-Sak Leung,et al.  Using Evolutionary Programming and Minimum Description Length Principle for Data Mining of Bayesian Networks , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[58]  Pedro Larrañaga,et al.  Structure Learning of Bayesian Networks by Genetic Algorithms: A Performance Analysis of Control Parameters , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[59]  Fred W. Glover,et al.  Tabu Search - Part I , 1989, INFORMS J. Comput..

[60]  Luis M. de Campos,et al.  Learning Causal Polytrees , 1993, ECSQARU.

[61]  Gregory F. Cooper,et al.  An Entropy-driven System for Construction of Probabilistic Expert Systems from Databases , 1990, UAI.

[62]  Fred Glover,et al.  Tabu Search - Part II , 1989, INFORMS J. Comput..

[63]  Jose Miguel Puerta,et al.  An iterated local search algorithm for learning Bayesian networks with restarts based on conditional independence tests , 2003, Int. J. Intell. Syst..

[64]  D. Madigan,et al.  Model Selection and Accounting for Model Uncertainty in Graphical Models Using Occam's Window , 1994 .

[65]  Moninder Singh,et al.  An Algorithm for the Construction of Bayesian Network Structures from Data , 1993, UAI.

[66]  Wray L. Buntine.  Theory Refinement on Bayesian Networks , 1991, UAI.

[67]  Wray L. Buntine.  A Guide to the Literature on Learning Probabilistic Networks from Data , 1996, IEEE Trans. Knowl. Data Eng..

[68]  Christopher Meek,et al.  Causal inference and causal explanation with background knowledge , 1995, UAI.

[69]  José A. Gámez,et al.  Learning Bayesian networks by Ant Colony Optimisation: searching in two different spaces , 2002 .

[70]  Nir Friedman,et al.  Learning Bayesian Networks with Local Structure , 1996, UAI.

[71]  Jose Miguel Puerta,et al.  Ant colony optimization for learning Bayesian networks , 2002, Int. J. Approx. Reason..

[72]  Dan Geiger,et al.  Learning Causal Trees from Dependence Information , 1990, AAAI.

[73]  Remco R. Bouckaert,et al.  Probalistic Network Construction Using the Minimum Description Length Principle , 1993, ECSQARU.