A Review of Inference Algorithms for Hybrid Bayesian Networks

Hybrid Bayesian networks have received an increasing attention during the last years. The difference with respect to standard Bayesian networks is that they can host discrete and continuous variables simultaneously, which extends the applicability of the Bayesian network framework in general. However, this extra feature also comes at a cost: inference in these types of models is computationally more challenging and the underlying models and updating procedures may not even support closed-form solutions. In this paper we provide an overview of the main trends and principled approaches for performing inference in hybrid Bayesian networks. The methods covered in the paper are organized and discussed according to their methodological basis. We consider how the methods have been extended and adapted to also include (hybrid) dynamic Bayesian networks, and we end with an overview of established software systems supporting inference in these types of models.

[1]  Manfred Jaeger,et al.  Probabilistic Decision Graphs - Combining Verification And Ai Techniques For Probabilistic Inference , 2002, Int. J. Uncertain. Fuzziness Knowl. Based Syst..

[2]  Rafael Rumí,et al.  Inference in hybrid Bayesian networks with Mixtures of Truncated Basis Functions , 2012, PGM 2012.

[3]  N. Wermuth,et al.  Graphical Models for Associations between Variables, some of which are Qualitative and some Quantitative , 1989 .

[4]  Zhaoyu Li,et al.  Efficient inference in Bayes networks as a combinatorial optimization problem , 1994, Int. J. Approx. Reason..

[5]  Andrés R. Masegosa,et al.  Parallel Importance Sampling in Conditional Linear Gaussian Networks , 2015, CAEPIA.

[6]  Prakash P. Shenoy.  Inference in Hybrid Bayesian Networks Using Mixtures of Gaussians , 2006, UAI.

[7]  David J. Spiegelhalter,et al.  Local computations with probabilities on graphical structures and their application to expert systems , 1990 .

[8]  Rafael Rumí,et al.  Supervised Classification Using Hybrid Probabilistic Decision Graphs , 2014, Probabilistic Graphical Models.

[9]  Keiji Kanazawa,et al.  A model for reasoning about persistence and causation , 1989 .

[10]  Michael I. Jordan,et al.  Loopy Belief Propagation for Approximate Inference: An Empirical Study , 1999, UAI.

[11]  Tom Minka,et al.  Expectation Propagation for approximate Bayesian inference , 2001, UAI.

[12]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Andrés R. Masegosa,et al.  Scaling up Bayesian variational inference using distributed computing clusters , 2017, Int. J. Approx. Reason..

[14]  Mark A. Paskin.  Sample Propagation , 2003, NIPS.

[15]  Prakash P. Shenoy,et al.  Probability propagation , 1990, Annals of Mathematics and Artificial Intelligence.

[16]  Zoubin Ghahramani,et al.  Propagation Algorithms for Variational Bayesian Learning , 2000, NIPS.

[17]  W. Freeman,et al.  Bethe free energy, Kikuchi approximations, and belief propagation algorithms , 2001 .

[18]  Jaswinder Pal Singh,et al.  A parallel Lauritzen-Spiegelhalter algorithm for probabilistic inference , 1994, Proceedings of Supercomputing '94.

[19]  V. Šmídl,et al.  The Variational Bayes Method in Signal Processing , 2005 .

[20]  Prakash P. Shenoy,et al.  Arc reversals in hybrid Bayesian networks with deterministic variables , 2009, Int. J. Approx. Reason..

[21]  Charles M. Bishop,et al.  Variational Message Passing , 2005, J. Mach. Learn. Res..

[22]  Uri Lerner,et al.  Inference in Hybrid Networks: Theoretical Limits and Practical Algorithms , 2001, UAI.

[23]  Volker Tresp,et al.  Model-independent mean-field theory as a local method for approximate propagation of information. , 1999, Network.

[24]  Sheila A. McIlraith,et al.  Monitoring a Complez Physical System using a Hybrid Dynamic Bayes Net , 2002, UAI.

[25]  Anders L. Madsen,et al.  Parallelization of Inference in Bayesian Networks , 1999 .

[26]  Prakash P. Shenoy,et al.  Operations for inference in continuous Bayesian networks with linear deterministic variables , 2006, Int. J. Approx. Reason..

[27]  Daniel Straub,et al.  Stochastic Modeling of Deterioration Processes through Dynamic Bayesian Networks , 2009 .

[28]  Junichi Mori,et al.  Inference in hybrid Bayesian networks with large discrete and continuous domains , 2016, Expert Syst. Appl..

[29]  Michael I. Jordan,et al.  Bayesian parameter estimation via variational methods , 2000, Stat. Comput..

[30]  Nevin Lianwen Zhang,et al.  Exploiting Causal Independence in Bayesian Network Inference , 1996, J. Artif. Intell. Res..

[31]  Kevin P. Murphy,et al.  Inference and Learning in Hybrid Bayesian Networks , 1998 .

[32]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[33]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[34]  Mohammad Modarres,et al.  Hybrid DBN monitoring and anomaly detection algorithms for on-line SHM , 2015, 2015 Annual Reliability and Maintainability Symposium (RAMS).

[35]  Avi Pfeffer,et al.  Object-Oriented Bayesian Networks , 1997, UAI.

[36]  Sanyang Liu,et al.  Propagation in CLG Bayesian networks based on semantic modeling , 2011, Artificial Intelligence Review.

[37]  Yuan Qi,et al.  Parameter Expanded Variational Bayesian Methods , 2006, NIPS.

[38]  Matthew Collette,et al.  A dynamic discretization method for reliability inference in Dynamic Bayesian Networks , 2015, Reliab. Eng. Syst. Saf..

[39]  Olivier Pourret,et al.  Bayesian networks : a practical guide to applications , 2008 .

[40]  William T. Freeman,et al.  Nonparametric belief propagation , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[41]  Uffe Kjaerulff,et al.  A computational scheme for Reasoning in Dynamic Probabilistic Networks , 2013, 1303.5407.

[42]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[43]  W. K. Hastings,et al.  Monte Carlo Sampling Methods Using Markov Chains and Their Applications , 1970 .

[44]  W. Wiegerinck,et al.  Approximate inference techniques with expectation constraints , 2005 .

[45]  David Barber,et al.  Expectation Correction for Smoothed Inference in Switching Linear Dynamical Systems , 2006, J. Mach. Learn. Res..

[46]  Oswaldo Morales-Nápoles,et al.  Non-parametric Bayesian networks: Improving theory and reviewing applications , 2015, Reliab. Eng. Syst. Saf..

[47]  Anders L. Madsen,et al.  LAZY Propagation: A Junction Tree Inference Algorithm Based on Lazy Evaluation , 1999, Artif. Intell..

[48]  Wei Sun,et al.  Probabilistic inference using linear Gaussian importance sampling for hybrid Bayesian networks , 2005, SPIE Defense + Commercial Sensing.

[49]  Vibhav Gogate,et al.  Approximate Inference Algorithms for Hybrid Bayesian Networks with Discrete Constraints , 2005, UAI.

[50]  S. Ghosal,et al.  Kullback Leibler property of kernel mixture priors in Bayesian density estimation , 2007, 0710.2746.

[51]  Hagai Attias,et al.  A Variational Bayesian Framework for Graphical Models , 1999 .

[52]  Adnan Darwiche,et al.  A differential approach to inference in Bayesian networks , 2000, JACM.

[53]  Matthew J. Beal.  Variational algorithms for approximate Bayesian inference , 2003 .

[54]  Nir Friedman,et al.  Bayesian Network Classifiers , 1997, Machine Learning.

[55]  Padhraic Smyth,et al.  Markov monitoring with unknown states , 1994, IEEE J. Sel. Areas Commun..

[56]  Rafael Rumí,et al.  Answering queries in hybrid Bayesian networks using importance sampling , 2012, Decis. Support Syst..

[57]  Serafín Moral,et al.  Approximating Conditional MTE Distributions by Means of Mixed Trees , 2003, ECSQARU.

[58]  Rafael Rumí,et al.  Approximate probability propagation with mixtures of truncated exponentials , 2007, Int. J. Approx. Reason..

[59]  Jeffrey K. Uhlmann,et al.  New extension of the Kalman filter to nonlinear systems , 1997, Defense, Security, and Sensing.

[60]  Rafael Rumí,et al.  Penniless Propagation with Mixtures of Truncated Exponentials , 2005, ECSQARU.

[61]  Viktor K. Prasanna,et al.  Scalable parallel implementation of exact inference in Bayesian networks , 2006, 12th International Conference on Parallel and Distributed Systems - (ICPADS'06).

[62]  Chong Wang,et al.  Stochastic variational inference , 2012, J. Mach. Learn. Res..

[63]  Xavier Boyen,et al.  Tractable Inference for Complex Stochastic Processes , 1998, UAI.

[64]  Changhe Yuan,et al.  Importance Sampling for General Hybrid Bayesian Networks , 2007, AISTATS.

[65]  Yee Whye Teh,et al.  The Infinite Factorial Hidden Markov Model , 2008, NIPS.

[66]  Joshua B. Tenenbaum,et al.  Infinite Dynamic Bayesian Networks , 2011, ICML.

[67]  Kevin P. Murphy,et al.  A Variational Approximation for Bayesian Networks with Discrete and Continuous Latent Variables , 1999, UAI.

[68]  Stuart J. Russell,et al.  Dynamic bayesian networks: representation, inference and learning , 2002 .

[69]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine Learning.

[70]  Matthew P. Wand,et al.  Kernel Smoothing , 1995 .

[71]  Tom Heskes,et al.  Generalized belief propagation for approximate inference in hybrid Bayesian networks , 2003, AISTATS.

[72]  Serafín Moral,et al.  Penniless propagation in join trees , 2000, Int. J. Intell. Syst..

[73]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[74]  Uri Lerner,et al.  Exact Inference in Networks with Discrete Children of Continuous Parents , 2001, UAI.

[75]  Nando de Freitas,et al.  Rao-Blackwellised Particle Filtering for Dynamic Bayesian Networks , 2000, UAI.

[76]  Rafael Rumí,et al.  Aalborg Universitet Inference in hybrid Bayesian networks , 2016 .

[77]  Andre Wibisono,et al.  Streaming Variational Bayes , 2013, NIPS.

[78]  Prakash P. Shenoy,et al.  Practical Aspects of Solving Hybrid Bayesian Networks Containing Deterministic Conditionals , 2015, Int. J. Intell. Syst..

[79]  Christophe Gonzales,et al.  On conditional truncated densities Bayesian networks , 2018, Int. J. Approx. Reason..

[80]  Richard E. Turner,et al.  Two problems with variational expectation maximisation for time-series models , 2011 .

[81]  Marek J Druzdzel,et al.  Canonical Probabilistic Models for Knowledge Engineering , 2007 .

[82]  Chonlagarn Iamsumang,et al.  Computational algorithm for dynamic hybrid Bayesian network in on-line system health management applications , 2014, 2014 International Conference on Prognostics and Health Management.

[83]  Michael Luby,et al.  Approximating Probabilistic Inference in Bayesian Belief Networks is NP-Hard , 1993, Artif. Intell..

[84]  Dragomir Anguelov,et al.  A General Algorithm for Approximate Inference and Its Application to Hybrid Bayes Nets , 1999, UAI.

[85]  R. van Engelen,et al.  Approximating Bayesian belief networks by arc removal , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[86]  Andrés R. Masegosa,et al.  d-VMP: Distributed Variational Message Passing , 2016, Probabilistic Graphical Models.

[87]  Michael I. Jordan,et al.  Factorial Hidden Markov Models , 1995, Machine Learning.

[88]  Adnan Darwiche,et al.  Recursive conditioning , 2001, Artif. Intell..

[89]  Sean Gerrish,et al.  Black Box Variational Inference , 2013, AISTATS.

[90]  Prakash P. Shenoy,et al.  Inference in hybrid Bayesian networks using mixtures of polynomials , 2011, Int. J. Approx. Reason..

[91]  Kuo-Chu Chang,et al.  Weighing and Integrating Evidence for Stochastic Simulation in Bayesian Networks , 2013, UAI.

[92]  Martin Neil,et al.  Building large-scale Bayesian networks , 2000, The Knowledge Engineering Review.

[93]  Michael I. Jordan,et al.  Variational Bayesian Inference with Stochastic Search , 2012, ICML.

[94]  David M. Pennock.  Logarithmic Time Parallel Bayesian Inference , 1998, UAI.

[95]  Ross D. Shachter,et al.  Fusion and Propagation with Multiple Observations in Belief Networks , 1991, Artif. Intell..

[96]  Serafín Moral,et al.  Mixtures of Truncated Exponentials in Hybrid Bayesian Networks , 2001, ECSQARU.

[97]  Jaakko Luttinen,et al.  Fast Variational Bayesian Linear State-Space Model , 2013, ECML/PKDD.

[98]  Robert G. Cowell,et al.  Local Propagation in Conditional Gaussian Bayesian Networks , 2005, J. Mach. Learn. Res..

[99]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[100]  Kuo-Chu Chang,et al.  Scalable inference for hybrid Bayesian networks with full density estimations , 2010, 2010 13th International Conference on Information Fusion.

[101]  Rafael Rumí,et al.  Mixtures of truncated basis functions , 2012, Int. J. Approx. Reason..

[102]  Xuan Vinh Nguyen,et al.  Data Discretization for Dynamic Bayesian Network Based Modeling of Genetic Networks , 2012, ICONIP.

[103]  Prakash P. Shenoy,et al.  Inference in hybrid Bayesian networks with mixtures of truncated exponentials , 2006, Int. J. Approx. Reason..

[104]  Gregory F. Cooper,et al.  The Computational Complexity of Probabilistic Inference Using Bayesian Belief Networks , 1990, Artif. Intell..

[105]  Geoffrey E. Hinton,et al.  Variational Learning for Switching State-Space Models , 2000, Neural Computation.

[106]  Marco Grzegorczyk,et al.  Nonparametric Bayesian Networks , 2011 .

[107]  Steffen L. Lauritzen,et al.  Stable local computation with conditional Gaussian distributions , 2001, Stat. Comput..

[108]  Xavier Boyen,et al.  Exploiting the Architecture of Dynamic Systems , 1999, AAAI/IAAI.

[109]  Srinivas Aluru,et al.  A parallel algorithm for exact Bayesian network inference , 2009, 2009 International Conference on High Performance Computing (HiPC).

[110]  A. Salmerón,et al.  Importance sampling in Bayesian networks using probability trees , 2000 .

[111]  Yoshua Bengio,et al.  Markovian Models for Sequential Data , 2004 .

[112]  Prakash P. Shenoy,et al.  Extended Shenoy-Shafer architecture for inference in hybrid bayesian networks with deterministic conditionals , 2011, Int. J. Approx. Reason..

[113]  José Mira,et al.  DISTRIBUTED INFERENCE IN BAYESIAN NETWORKS , 1994 .

[114]  Bo Thiesson,et al.  Learning Mixtures of DAG Models , 1998, UAI.

[115]  Daphne Koller,et al.  Continuous Time Bayesian Networks , 2012, UAI.

[116]  Andrew W. Moore,et al.  Mix-nets: Factored Mixtures of Gaussians in Bayesian Networks with Mixed Continuous And Discrete Variables , 2000, UAI.

[117]  Prakash P. Shenoy,et al.  Axioms for probability and belief-function proagation , 1990, UAI.

[118]  Uffe Kjaerulff.  Reduction of Computational Complexity in Bayesian Networks through Removal of Weak Dependencies , 2011 .

[119]  HrycejTomas.  Gibbs sampling in Bayesian networks (research note) , 1990 .

[120]  Anders L. Madsen,et al.  Belief update in CLG Bayesian networks with lazy propagation , 2006, Int. J. Approx. Reason..

[121]  José A. Gámez,et al.  Modelling and inference with Conditional Gaussian Probabilistic Decision Graphs , 2012, Int. J. Approx. Reason..

[122]  S. Lauritzen.  Propagation of Probabilities, Means, and Variances in Mixed Graphical Association Models , 1992 .

[123]  Jaswinder Pal Singh,et al.  Parallel Implementations of Probabilistic Inference , 1996, Computer.

[124]  Pedro Larrañaga,et al.  Bayesian classifiers based on kernel density estimation: Flexible classifiers , 2009, Int. J. Approx. Reason..

[125]  Changhe Yuan,et al.  Generalized Evidence Pre-propagated Importance Sampling for Hybrid Bayesian Networks , 2007, AAAI.

[126]  Daphne Koller,et al.  Nonuniform Dynamic Discretization in Hybrid Networks , 1997, UAI.