Bayesian Networks

Publisher Summary A Bayesian network is a tool for modeling and reasoning with uncertain beliefs; it comprises two parts: a qualitative component in the form of a directed acyclic graph (DAG) and a quantitative component in the form conditional probabilities. Intuitively, the DAG of a Bayesian network explicates variables of interest (DAG nodes) and the direct influences among them (DAG edges). The conditional probabilities of a Bayesian network quantify the dependencies between variables and their parents in the DAG. Formally though, a Bayesian network is interpreted as specifying a unique probability distribution over its variables. Hence, the network can be viewed as a factored (compact) representation of an exponentially sized probability distribution. The formal syntax and semantics of Bayesian networks are discussed in this chapter. The power of Bayesian networks as a representational tool stems both from the ability to represent large probability distributions compactly and the availability of inference algorithms to answer queries about these distributions without necessarily constructing them explicitly. The chapter also discusses exact inference algorithms and approximate inference algorithms.

[1]  H. Bozdogan.  Model selection and Akaike's Information Criterion (AIC): The general theory and its analytical extensions , 1987 .

[2]  David Heckerman,et al.  A Tutorial on Learning with Bayesian Networks , 1999, Innovations in Bayesian Networks.

[3]  Adnan Darwiche,et al.  Any-Space Probabilistic Inference , 2000, UAI.

[4]  Adnan Darwiche,et al.  Solving MAP Exactly using Systematic Search , 2002, UAI.

[5]  Kevin P. Murphy,et al.  Space-Efficient Inference in Dynamic Probabilistic Networks , 1997, IJCAI.

[6]  Rina Dechter,et al.  Topological parameters for time-space tradeoff , 1996, Artif. Intell..

[7]  Keiji Kanazawa,et al.  A model for reasoning about persistence and causation , 1989 .

[8]  Dan Geiger,et al.  Identifying independence in bayesian networks , 1990, Networks.

[9]  David Maxwell Chickering,et al.  Learning Bayesian Networks: The Combination of Knowledge and Statistical Data , 1994, Machine Learning.

[10]  S. Lauritzen.  The EM algorithm for graphical association models with missing data , 1995 .

[11]  Stuart J. Russell,et al.  Local Learning in Probabilistic Networks with Hidden Variables , 1995, IJCAI.

[12]  Linda C. van der Gaag,et al.  Making Sensitivity Analysis Computationally Efficient , 2000, UAI.

[13]  Gregory M. Provan,et al.  Knowledge Engineering for Large Belief Networks , 1994, UAI.

[14]  David Heckerman,et al.  Causal Independence for Knowledge Acquisition and Inference , 1993, UAI.

[15]  Henry A. Kautz,et al.  Solving Bayesian Networks by Weighted Model Counting , 2005 .

[16]  De Raedt,et al.  Advances in Inductive Logic Programming , 1996 .

[17]  Jorma Rissanen,et al.  Stochastic Complexity in Statistical Inquiry , 1989, World Scientific Series in Computer Science.

[18]  Enrique F. Castillo,et al.  Sensitivity analysis in discrete Bayesian networks , 1997, IEEE Trans. Syst. Man Cybern. Part A.

[19]  Adnan Darwiche,et al.  A differential approach to inference in Bayesian networks , 2000, JACM.

[20]  Tom Heskes,et al.  On the Uniqueness of Loopy Belief Propagation Fixed Points , 2004, Neural Computation.

[21]  Rina Dechter,et al.  Hybrid Processing of Beliefs and Constraints , 2001, UAI.

[22]  M. Opper,et al.  Advanced mean field methods: theory and practice , 2001 .

[23]  David Maxwell Chickering,et al.  Large-Sample Learning of Bayesian Networks is NP-Hard , 2002, J. Mach. Learn. Res..

[24]  Finn V. Jensen,et al.  Bayesian Networks and Decision Graphs , 2001, Statistics for Engineering and Information Science.

[25]  Sean R. Eddy,et al.  Biological Sequence Analysis: Probabilistic Models of Proteins and Nucleic Acids , 1998 .

[26]  Yee Whye Teh,et al.  Belief Optimization for Binary Networks: A Stable Alternative to Loopy Belief Propagation , 2001, UAI.

[27]  Finn Verner Jensen,et al.  Gradient Descent Training of Bayesian Networks , 1999, ESCQARU.

[28]  Steen Andreassen,et al.  MUNIN - A Causal Probabilistic Network for Interpretation of Electromyographic Findings , 1987, IJCAI.

[29]  Dan Geiger,et al.  Optimizing exact genetic linkage computations , 2003, RECOMB '03.

[30]  John S. Breese,et al.  CONSTRUCTION OF BELIEF AND DECISION NETWORKS , 1992, Comput. Intell..

[31]  Adnan Darwiche,et al.  A distance measure for bounding probabilistic belief change , 2002, Int. J. Approx. Reason..

[32]  Kevin T. Kelly,et al.  Discovering Causal Structure. , 1989 .

[33]  Nir Friedman,et al.  Learning Bayesian Networks with Local Structure , 1996, UAI.

[34]  Nevin L. Zhang,et al.  A simple approach to Bayesian network computations , 1994 .

[35]  Ben Taskar,et al.  Learning Probabilistic Models of Relational Structure , 2001, ICML.

[36]  Huaiyu Zhu.  On Information and Sufficiency , 1997 .

[37]  Max Welling,et al.  On the Choice of Regions for Generalized Belief Propagation , 2004, UAI.

[38]  Dan Roth,et al.  Lifted First-Order Probabilistic Inference , 2005, IJCAI.

[39]  S. Muggleton.  Stochastic Logic Programs , 1996 .

[40]  Peter Haddawy,et al.  Answering Queries from Context-Sensitive Probabilistic Knowledge Bases , 1997, Theor. Comput. Sci..

[41]  John E. Desmedt,et al.  Computer Aided Electromyography and Expert Systems , 1989 .

[42]  Dan Roth,et al.  On the Hardness of Approximate Reasoning , 1993, IJCAI.

[43]  David Allen,et al.  New Advances in Inference by Recursive Conditioning , 2002, UAI.

[44]  Michael I. Jordan,et al.  Exploiting Tractable Substructures in Intractable Networks , 1995, NIPS.

[45]  David Poole,et al.  The Independent Choice Logic for Modelling Multiple Agents Under Uncertainty , 1997, Artif. Intell..

[46]  Uffe Kjærulff,et al.  A Computational Scheme for Reasoning in Dynamic Probabilistic Networks , 1992, UAI.

[47]  Finn Verner Jensen,et al.  MUNIN: an expert EMG assistant , 1988 .

[48]  Peter Haddawy,et al.  Generating Bayesian Networks from Probablity Logic Knowledge Bases , 1994, UAI.

[49]  Ann E. Nicholson,et al.  The Data Association Problem when Monitoring Robot Vehicles Using Dynamic Belief Networks , 1992, ECAI.

[50]  Judea Pearl,et al.  Causal networks: semantics and expressiveness , 2013, UAI.

[51]  Eric Horvitz,et al.  Bounded Conditioning: Flexible Inference for Decisions under Scarce Resources , 2013, UAI 1989.

[52]  Francisco Javier Díez,et al.  Local Conditioning in Bayesian Networks , 1996, Artif. Intell..

[53]  Manfred Jaeger,et al.  Compiling relational Bayesian networks for exact inference , 2006, Int. J. Approx. Reason..

[54]  Jian Cheng,et al.  AIS-BN: An Adaptive Importance Sampling Algorithm for Evidential Reasoning in Large Bayesian Networks , 2000, J. Artif. Intell. Res..

[55]  David J. Spiegelhalter,et al.  Local computations with probabilities on graphical structures and their application to expert systems , 1990 .

[56]  Stuart J. Russell,et al.  Stochastic simulation algorithms for dynamic probabilistic networks , 1995, UAI.

[57]  Gregory F. Cooper,et al.  The Computational Complexity of Probabilistic Inference Using Bayesian Belief Networks , 1990, Artif. Intell..

[58]  Stefan Arnborg,et al.  Efficient algorithms for combinatorial problems on graphs with bounded decomposability — A survey , 1985, BIT.

[59]  Toniann Pitassi,et al.  Value Elimination: Bayesian Interence via Backtracking Search , 2002, UAI.

[60]  Lise Getoor,et al.  Learning Probabilistic Relational Models , 1999, IJCAI.

[61]  D. Heckerman,et al.  ,81. Introduction , 2022 .

[62]  Dan Geiger,et al.  d-Separation: From Theorems to Algorithms , 2013, UAI.

[63]  Steffen L. Lauritzen,et al.  Bayesian updating in causal probabilistic networks by local computations , 1990 .

[64]  Rina Dechter,et al.  Bucket Elimination: A Unifying Framework for Reasoning , 1999, Artif. Intell..

[65]  Enrico Macii,et al.  Algebraic decision diagrams and their applications , 1993, Proceedings of 1993 International Conference on Computer Aided Design (ICCAD).

[66]  Francisco Javier Díez,et al.  Parameter adjustment in Bayes networks. The generalized noisy OR-gate , 1993, UAI.

[67]  Kuo-Chu Chang,et al.  Weighing and Integrating Evidence for Stochastic Simulation in Bayesian Networks , 2013, UAI.

[68]  David Maier,et al.  The Theory of Relational Databases , 1983 .

[69]  Michael I. Jordan,et al.  Probabilistic Independence Networks for Hidden Markov Probability Models , 1997, Neural Computation.

[70]  Adnan Darwiche,et al.  A Logical Approach to Factoring Belief Networks , 2002, KR.

[71]  Prakash P. Shenoy,et al.  Propagating Belief Functions with Local Computations , 1986, IEEE Expert.

[72]  Solomon Eyal Shimony,et al.  Finding MAPs for Belief Networks is NP-Hard , 1994, Artif. Intell..

[73]  David Maxwell Chickering,et al.  Optimal Structure Identification With Greedy Search , 2003, J. Mach. Learn. Res..

[74]  G. McLachlan,et al.  The EM algorithm and extensions , 1996 .

[75]  Nicole A. Lazar,et al.  Statistical Analysis With Missing Data , 2003, Technometrics.

[76]  Ross D. Shachter,et al.  Symbolic Probabilistic Inference in Belief Networks , 1990, AAAI.

[77]  Kathryn B. Laskey,et al.  Network Fragments: Representing Knowledge for Constructing Probabilistic Models , 1997, UAI.

[78]  Wai Lam,et al.  LEARNING BAYESIAN BELIEF NETWORKS: AN APPROACH BASED ON THE MDL PRINCIPLE , 1994, Comput. Intell..

[79]  David Heckerman,et al.  A combination of cutset conditioning with clique-tree propagation in the Pathfinder system , 1990, UAI.

[80]  Ross D. Shachter.  Evidence Absorption and Propagation through Evidence Reversals , 2013, UAI.

[81]  Douglas W. Nychka,et al.  Discovering Causal Structure , 1989 .

[82]  Gregory F. Cooper,et al.  A randomized approximation algorithm for probabilistic inference on bayesian belief networks , 1990, Networks.

[83]  Adnan Darwiche,et al.  On the Revision of Probabilistic Beliefs using Uncertain Evidence , 2003, IJCAI.

[84]  Nir Friedman,et al.  On the Sample Complexity of Learning Bayesian Networks , 1996, UAI.

[85]  Tom Heskes,et al.  Fractional Belief Propagation , 2002, NIPS.

[86]  C. N. Liu,et al.  Approximating discrete probability distributions with dependence trees , 1968, IEEE Trans. Inf. Theory.

[87]  Russell Greiner,et al.  Discriminative Model Selection for Belief Net Structures , 2005, AAAI.

[88]  Rina Dechter,et al.  AND/OR Branch-and-Bound for Graphical Models , 2005, IJCAI.

[89]  Max Henrion,et al.  Some Practical Issues in Constructing Belief Networks , 1987, UAI.

[90]  F. Díez,et al.  An efficient factorization for the noisy MAX ∗ , 2007 .

[91]  Michael I. Jordan,et al.  Loopy Belief Propagation for Approximate Inference: An Empirical Study , 1999, UAI.

[92]  José A. Gámez,et al.  Advances in Bayesian networks , 2004 .

[93]  A. Darwiche,et al.  Complexity Results and Approximation Strategies for MAP Explanations , 2011, J. Artif. Intell. Res..

[94]  Judea Pearl,et al.  Fusion, Propagation, and Structuring in Belief Networks , 1986, Artif. Intell..

[95]  Prakash P. Shenoy,et al.  A Comparison of Lauritzen-Spiegelhalter, Hugin, and Shenoy-Shafer Architectures for Computing Marginals of Probability Distributions , 1998, UAI.

[96]  Rina Dechter,et al.  Bayesian Inference in the Presence of Determinism , 2003, AISTATS.

[97]  Kathryn B. Laskey.  Sensitivity analysis for probability assessments in Bayesian networks , 1995, IEEE Trans. Syst. Man Cybern..

[98]  Francisco Javier Díez,et al.  Efficient computation for the noisy MAX , 2003, Int. J. Intell. Syst..

[99]  Gregory F. Cooper,et al.  A Bayesian method for the induction of probabilistic networks from data , 1992, Machine Learning.

[100]  Volker Tresp,et al.  Model-independent mean-field theory as a local method for approximate propagation of information. , 1999, Network.

[101]  Avi Pfeffer,et al.  Object-Oriented Bayesian Networks , 1997, UAI.

[102]  David Allen,et al.  Exploiting Evidence in Probabilistic Inference , 2005, UAI.

[103]  S. F. Galán,et al.  An e-cient factorization for the noisy MAX ⁄ , 2007 .

[104]  Nir Friedman,et al.  The Bayesian Structural EM Algorithm , 1998, UAI.

[105]  Jeff A. Bilmes,et al.  On Triangulating Dynamic Graphical Models , 2002, UAI.

[106]  Michael Luby,et al.  Approximating Probabilistic Inference in Bayesian Belief Networks is NP-Hard , 1993, Artif. Intell..

[107]  Rina Dechter,et al.  A general scheme for automatic generation of search heuristics from specification dependencies , 2001, Artif. Intell..

[108]  Randolph A. Miller,et al.  Using Causal Knowledge to Create Simulated Patient Cases: The CPCS Project as an Extension of INTERNIST-1 , 1988 .

[109]  Judea Pearl,et al.  Evidential Reasoning Using Stochastic Simulation of Causal Models , 1987, Artif. Intell..

[110]  Adnan Darwiche,et al.  Compiling Bayesian Networks with Local Structure , 2005, IJCAI.

[111]  Michael I. Jordan,et al.  A generalized mean field algorithm for variational inference in exponential families , 2002, UAI.

[112]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[113]  Sampath Srinivas,et al.  A Generalization of the Noisy-Or Model , 1993, UAI.

[114]  Nevin Lianwen Zhang,et al.  Exploiting Causal Independence in Bayesian Network Inference , 1996, J. Artif. Intell. Res..

[115]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[116]  Taisuke Sato,et al.  A Statistical Learning Method for Logic Programs with Distribution Semantics , 1995, ICLP.

[117]  Yoshua Bengio,et al.  Pattern Recognition and Neural Networks , 1995 .

[118]  F E Masarie,et al.  Quick medical reference (QMR) for diagnostic assistance. , 1986, M.D.Computing.

[119]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[120]  David Poole,et al.  First-order probabilistic inference , 2003, IJCAI.

[121]  Rina Dechter,et al.  Mixtures of Deterministic-Probabilistic Networks and their AND/OR Search Space , 2004, UAI.

[122]  Alan L. Yuille,et al.  CCCP Algorithms to Minimize the Bethe and Kikuchi Free Energies: Convergent Alternatives to Belief Propagation , 2002, Neural Computation.

[123]  Yuan Qi,et al.  Tree-structured Approximations by Expectation Propagation , 2003, NIPS.

[124]  Michael I. Jordan,et al.  Factorial Hidden Markov Models , 1995, Machine Learning.

[125]  Adnan Darwiche,et al.  Recursive conditioning , 2001, Artif. Intell..

[126]  David J. Spiegelhalter,et al.  Probabilistic Networks and Expert Systems , 1999, Information Science and Statistics.

[127]  S. Aji,et al.  The Generalized Distributive Law and Free Energy Minimization , 2001 .

[128]  Jin Tian,et al.  A Branch-and-Bound Algorithm for MDL Learning Bayesian Networks , 2000, UAI.

[129]  Johan Håstad,et al.  Tensor Rank is NP-Complete , 1989, ICALP.

[130]  Mark Hopkins,et al.  Using Recursive Decomposition to Construct Elimination Orders, Jointrees, and Dtrees , 2001, ECSQARU.

[131]  Robert J. McEliece,et al.  Belief Propagation on Partially Ordered Sets , 2003, Mathematical Systems Theory in Biology, Communications, Computation, and Finance.

[132]  Derek G. Corneil,et al.  Complexity of finding embeddings in a k -tree , 1987 .

[133]  Adnan Darwiche,et al.  Morphing the Hugin and Shenoy-Shafer Architectures , 2003, ECSQARU.

[134]  Adnan Darwiche,et al.  A differential semantics for jointree algorithms , 2002, Artif. Intell..

[135]  Adnan Darwiche,et al.  Sensitivity Analysis in Bayesian Networks: From Single to Multiple Parameters , 2004, UAI.

[136]  Craig Boutilier,et al.  Context-Specific Independence in Bayesian Networks , 1996, UAI.

[137]  Luc De Raedt,et al.  Towards Combining Inductive Logic Programming with Bayesian Networks , 2001, ILP.

[138]  Tom Heskes,et al.  Stable Fixed Points of Loopy Belief Propagation Are Local Minima of the Bethe Free Energy , 2002, NIPS.

[139]  Patrick Brézillon,et al.  Lecture Notes in Artificial Intelligence , 1999 .

[140]  Ross D. Shachter,et al.  Global Conditioning for Probabilistic Inference in Belief Networks , 1994, UAI.

[141]  Ross D. Shachter.  Evaluating Influence Diagrams , 1986, Oper. Res..

[142]  Yee Whye Teh,et al.  Structured Region Graphs: Morphing EP into GBP , 2005, UAI.

[143]  Ross D. Shachter,et al.  Simulation Approaches to General Probabilistic Inference on Belief Networks , 2013, UAI.

[144]  Andrew P. Sage,et al.  Uncertainty in Artificial Intelligence , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[145]  Changhe Yuan,et al.  An Importance Sampling Algorithm Based on Evidence Pre-propagation , 2002, UAI.

[146]  Brendan J. Frey,et al.  A Revolution: Belief Propagation in Graphs with Cycles , 1997, NIPS.

[147]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[148]  Judea Pearl,et al.  Bayesian Networks , 1998, Encyclopedia of Social Network Analysis and Mining. 2nd Ed..

[149]  Paul D. Seymour,et al.  Graph Minors. II. Algorithmic Aspects of Tree-Width , 1986, J. Algorithms.

[150]  Max Henrion,et al.  Propagating uncertainty in bayesian networks by probabilistic logic sampling , 1986, UAI.

[151]  Manfred Jaeger,et al.  Relational Bayesian Networks , 1997, UAI.

[152]  Adnan Darwiche,et al.  When do Numbers Really Matter? , 2001, UAI.

[153]  Steen Andreassen,et al.  Improving the Diagnostic Performance of MUNIN by Remodelling of the Diseases , 2001, AIME.

[154]  David Allen,et al.  Optimal Time-Space Tradeoff in Probabilistic Inference , 2003, Probabilistic Graphical Models.

[155]  Nevin Lianwen Zhang,et al.  Exploiting Contextual Independence In Probabilistic Inference , 2011, J. Artif. Intell. Res..

[156]  J. Pearl.  Causality: Models, Reasoning and Inference , 2000 .

[157]  Dan Geiger,et al.  Exact genetic linkage computations for general pedigrees , 2002, ISMB.

[158]  Thomas D. Nielsen,et al.  Using ROBDDs for Inference in Bayesian Networks with Troubleshooting as an Example , 2000, UAI.

[159]  Petr Savický,et al.  Tensor rank-one decomposition of probability tables , 2005 .

[160]  Yan Lin,et al.  Computational Advantages of Relevance Reasoning in Bayesian Belief Networks , 1997, UAI.

[161]  Rina Dechter,et al.  Iterative Join-Graph Propagation , 2002, UAI.

[162]  Jeremy York,et al.  Use of the Gibbs Sampler in Expert Systems , 1992, Artif. Intell..

[163]  Adnan Darwiche,et al.  Conditioning Algorithms for Exact and Approximate Inference in Causal Networks , 1995, UAI.

[164]  Hilbert J. Kappen,et al.  Novel iteration schemes for the Cluster Variation Method , 2001, NIPS.

[165]  Brendan J. Frey,et al.  Sequentially Fitting "Inclusive" Trees for Inference in Noisy-OR Networks , 2000, NIPS.

[166]  William T. Freeman,et al.  Comparison of graph cuts with belief propagation for stereo, using identical MRF parameters , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[167]  Robert P. Goldman,et al.  Dynamic construction of belief networks , 1990, UAI.

[168]  Jirí Vomlel,et al.  Exploiting Functional Dependence in Bayesian Network Inference , 2002, UAI.

[169]  Jesse Hoey,et al.  SPUDD: Stochastic Planning using Decision Diagrams , 1999, UAI.

[170]  Reuven Bar-Yehuda,et al.  Random Algorithms for the Loop Cutset Problem , 1999, UAI.

[171]  Michael I. Jordan.  Learning in Graphical Models , 1999, NATO ASI Series.

[172]  Adnan Darwiche,et al.  Modeling and Reasoning with Bayesian Networks , 2009 .

[173]  Martin J. Wainwright,et al.  Tree-based reparameterization for approximate inference on loopy graphs , 2001, NIPS.

[174]  Stuart J. Russell,et al.  Approximate inference for first-order probabilistic languages , 2001, IJCAI.

[175]  David Poole,et al.  Context-specific approximation in probabilistic inference , 1998, UAI.

[176]  Joe Suzuki,et al.  Learning Bayesian Belief Networks Based on the MDL Principle : An Efficient Algorithm Using the Branch and Bound Technique , 1999 .

[177]  Robert P. Goldman,et al.  From knowledge bases to decision models , 1992, The Knowledge Engineering Review.

[178]  Adnan Darwiche,et al.  Inference in belief networks: A procedural guide , 1996, Int. J. Approx. Reason..

[179]  Christopher Meek,et al.  Structured Variational Inference Procedures and their Realizations , 2005, AISTATS.

[180]  Manfred Jaeger,et al.  On the complexity of inference about probabilistic relational models , 2000, Artif. Intell..

[181]  Adnan Darwiche,et al.  Compiling Bayesian Networks Using Variable Elimination , 2007, IJCAI.

[182]  William T. Freeman,et al.  Understanding belief propagation and its generalizations , 2003 .

[183]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[184]  David Poole,et al.  Probabilistic Horn Abduction and Bayesian Networks , 1993, Artif. Intell..

[185]  Matthew Richardson,et al.  Markov logic networks , 2006, Machine Learning.

[186]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine Learning.

[187]  Wim Wiegerinck,et al.  Variational Approximations between Mean Field Theory and the Junction Tree Algorithm , 2000, UAI.