Finding Non-Overlapping Clusters for Generalized Inference Over Graphical Models

Graphical models use graphs to compactly capture stochastic dependencies amongst a collection of random variables. Inference over graphical models corresponds to finding marginal probability distributions given joint probability distributions. In general, this is computationally intractable, which has led to a quest for finding efficient approximate inference algorithms. We propose a framework for generalized inference over graphical models that can be used as a wrapper for improving the estimates of approximate inference algorithms. Instead of applying an inference algorithm to the original graph, we apply the inference algorithm to a block-graph, defined as a graph in which the nodes are non-overlapping clusters of nodes from the original graph. This results in marginal estimates of a cluster of nodes, which we further marginalize to get the marginal estimates of each node. Our proposed block-graph construction algorithm is simple, efficient, and motivated by the observation that approximate inference is more accurate on graphs with longer cycles. We present extensive numerical simulations that illustrate our block-graph framework with a variety of inference algorithms (e.g., those in the libDAI software package). These simulations show the improvements provided by our framework.

[1]  Dmitry M. Malioutov,et al.  Approximate inference in Gaussian graphical models , 2008 .

[2]  Martin J. Wainwright,et al.  Tree-based reparameterization framework for analysis of sum-product and related algorithms , 2003, IEEE Trans. Inf. Theory.

[3]  José M. F. Moura,et al.  Telescoping Recursive Representations and Estimation of Gauss–Markov Random Fields , 2009, IEEE Transactions on Information Theory.

[4]  Joris M. Mooij,et al.  libDAI: A Free and Open Source C++ Library for Discrete Approximate Inference in Graphical Models , 2010, J. Mach. Learn. Res..

[5]  Nikhil Balram,et al.  Recursive structure of noncausal Gauss-Markov random fields , 1992, IEEE Trans. Inf. Theory.

[6]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[7]  Arie M. C. A. Koster,et al.  Treewidth computations I. Upper bounds , 2010, Inf. Comput..

[8]  Paul D. Seymour,et al.  Graph Minors. II. Algorithmic Aspects of Tree-Width , 1986, J. Algorithms.

[9]  Fabio Gagliardi Cozman,et al.  Bucket-Tree Elimination for Automated Reasoning , 2001 .

[10]  David J. Spiegelhalter,et al.  Local computations with probabilities on graphical structures and their application to expert systems , 1990 .

[11]  Michael I. Jordan,et al.  Graph Partition Strategies for Generalized Mean Field Inference , 2004, UAI.

[12]  Max Welling,et al.  On the Choice of Regions for Generalized Belief Propagation , 2004, UAI.

[13]  Hilbert J. Kappen,et al.  Approximate Inference and Constrained Optimization , 2002, UAI.

[14]  Alessandro Pelizzola,et al.  Cluster Variation Method in Statistical Physics and Probabilistic Graphical Models , 2005, ArXiv.

[15]  Zoubin Ghahramani,et al.  Choosing a Variable to Clamp: Approximate Inference Using Conditioned Belief Propagation , 2009 .

[16]  W. Freeman,et al.  Generalized Belief Propagation , 2000, NIPS.

[17]  Hilbert J. Kappen,et al.  Loop Corrections for Approximate Inference on Factor Graphs , 2007, J. Mach. Learn. Res..

[18]  Gregory F. Cooper,et al.  NESTOR: A Computer-Based Medical Diagnostic Aid That Integrates Causal and Probabilistic Knowledge. , 1984 .

[19]  大西 仁,et al.  Pearl, J. (1988, second printing 1991). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan-Kaufmann. , 1994 .

[20]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[21]  Martin J. Wainwright,et al.  Stochastic processes on graphs with cycles: geometric and variational approaches , 2002 .

[22]  Tony Jebara,et al.  MAP Estimation, Message Passing, and Perfect Graphs , 2009, UAI.

[23]  Yun Peng,et al.  Plausibility of Diagnostic Hypotheses: The Nature of Simplicity , 1986, AAAI.

[24]  JOHN w. WOODS,et al.  Kalman filtering in two dimensions , 1977, IEEE Trans. Inf. Theory.

[25]  Steffen L. Lauritzen,et al.  Bayesian updating in causal probabilistic networks by local computations , 1990 .

[26]  Rina Dechter,et al.  Bucket Elimination: A Unifying Framework for Reasoning , 1999, Artif. Intell..

[27]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine Learning.

[28]  J. Besag.  Spatial Interaction and the Statistical Analysis of Lattice Systems , 1974 .

[29]  Prakash P. Shenoy,et al.  Probability propagation , 1990, Annals of Mathematics and Artificial Intelligence.

[30]  Martin J. Wainwright,et al.  MAP estimation via agreement on (hyper)trees: Message-passing and linear programming , 2005, ArXiv.

[31]  Trevor Hastie,et al.  Applications of the lasso and grouped lasso to the estimation of sparse graphical models , 2010 .

[32]  W. Freeman,et al.  Bethe free energy, Kikuchi approximations, and belief propagation algorithms , 2001 .

[33]  Yuan Qi,et al.  Tree-structured Approximations by Expectation Propagation , 2003, NIPS.

[34]  E. Fabre,et al.  Dealing with short cycles in graphical codes , 2000, 2000 IEEE International Symposium on Information Theory (Cat. No.00CH37060).

[35]  Martin J. Wainwright,et al.  MAP estimation via agreement on trees: message-passing and linear programming , 2005, IEEE Transactions on Information Theory.

[36]  Kevin P. Murphy,et al.  Sparse Gaussian graphical models with unknown block structure , 2009, ICML '09.

[37]  Nevin L. Zhang,et al.  A simple approach to Bayesian network computations , 1994 .

[38]  Larry A. Wasserman,et al.  Time varying undirected graphs , 2008, Machine Learning.

[39]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[40]  R. E. Kalman,et al.  New Results in Linear Filtering and Prediction Theory , 1961 .

[41]  Divyanshu Vats,et al.  Tree-Structured Like Representations for Continuous and Graph Indexed Markov Random Fields , 2011 .

[42]  Vibhav Gogate,et al.  Join-Graph Propagation Algorithms , 2010, J. Artif. Intell. Res..

[43]  Gregory F. Cooper,et al.  The Computational Complexity of Probabilistic Inference Using Bayesian Belief Networks , 1990, Artif. Intell..

[44]  Vladimir Kolmogorov,et al.  Convergent Tree-Reweighted Message Passing for Energy Minimization , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  H. Künsch.  Gaussian Markov random fields , 1979 .

[46]  Martin J. Wainwright,et al.  A new class of upper bounds on the log partition function , 2002, IEEE Transactions on Information Theory.

[47]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[48]  B. Bollobás.  The evolution of random graphs , 1984 .

[49]  Frank Jensen,et al.  Optimal junction Trees , 1994, UAI.

[50]  Uue Kjjrull.  Triangulation of Graphs { Algorithms Giving Small Total State Space Triangulation of Graphs { Algorithms Giving Small Total State Space , 1990 .

[51]  Dan Geiger,et al.  A sufficiently fast algorithm for finding close to optimal junction trees , 1996, UAI.

[52]  P. Levy.  A Special Problem of Brownian Motion, and a General Theory of Gaussian Random Functions , 1956 .

[53]  Yee Whye Teh,et al.  Structured Region Graphs: Morphing EP into GBP , 2005, UAI.

[54]  P. Erdos,et al.  On the evolution of random graphs , 1984 .

[55]  C. Striebel,et al.  On the maximum likelihood estimates for linear dynamic systems , 1965 .

[56]  Michael I. Jordan,et al.  A generalized mean field algorithm for variational inference in exponential families , 2002, UAI.

[57]  X. Jin.  Factor graphs and the Sum-Product Algorithm , 2002 .

[58]  H. Markowitz.  The Elimination form of the Inverse and its Application to Linear Programming , 1957 .

[59]  Ali Jalali,et al.  On Learning Discrete Graphical Models using Group-Sparse Regularization , 2011, AISTATS.

[60]  Qiang Liu,et al.  Bounding the Partition Function using Holder's Inequality , 2011, ICML.

[61]  R. Kikuchi.  A Theory of Cooperative Phenomena , 1951 .

[62]  Pinar Heggernes,et al.  The Minimum Degree Heuristic and the Minimal Triangulation Process , 2003, WG.

[63]  Tommi S. Jaakkola,et al.  Tree Block Coordinate Descent for MAP in Graphical Models , 2009, AISTATS.

[64]  Le Song,et al.  Estimating time-varying networks , 2008, ISMB 2008.

[65]  R. Halin.  S-functions for graphs , 1976 .

[66]  Brendan J. Frey,et al.  Factor graphs and the sum-product algorithm , 2001, IEEE Trans. Inf. Theory.

[67]  Sang-il Oum,et al.  Rank‐width of random graphs , 2010, J. Graph Theory.

[68]  A. Willsky,et al.  Solution and linear estimation of 2-D nearest-neighbor models , 1990 .

[69]  Martin J. Wainwright,et al.  Embedded trees: estimation of Gaussian Processes on graphs with cycles , 2004, IEEE Transactions on Signal Processing.

[70]  A. Montanari,et al.  How to compute loop corrections to the Bethe approximation , 2005, cond-mat/0506769.

[71]  D. West.  Introduction to Graph Theory , 1995 .

[72]  L. Held,et al.  Gaussian Markov Random Fields: Theory And Applications (Monographs on Statistics and Applied Probability) , 2005 .