Feedback message passing for inference in gaussian graphical models

For Gaussian graphical models with cycles, loopy belief propagation often performs reasonably well, but its convergence is not guaranteed and the computation of variances is generally incorrect. In this paper, we identify a set of special vertices called a feedback vertex set whose removal results in a cycle-free graph. We propose a feedback message passing algorithm in which non-feedback nodes send out one set of messages while the feedback nodes use a different message update scheme. Exact inference results can be obtained in O(k2n), where k is the number of feedback nodes and n is the total number of nodes. For graphs with large feedback vertex sets, we describe a tractable approximate feedback message passing algorithm. Experimental results show that this procedure converges more often, faster, and provides better results than loopy belief propagation.

[1]  Marco Grzegorczyk,et al.  Comparative evaluation of reverse engineering gene regulatory networks with relevance networks, graphical gaussian models and bayesian networks , 2006, Bioinform..

[2]  David Heckerman,et al.  Causal independence for probability assessment and inference using Bayesian networks , 1996, IEEE Trans. Syst. Man Cybern. Part A.

[3]  C. N. Liu,et al.  Approximating discrete probability distributions with dependence trees , 1968, IEEE Trans. Inf. Theory.

[4]  Piotr Berman,et al.  A 2-Approximation Algorithm for the Undirected Feedback Vertex Set Problem , 1999, SIAM J. Discret. Math..

[5]  E. Gaztañaga,et al.  Biasing and hierarchical statistics in large-scale structure , 1993, astro-ph/9302009.

[6]  Martin J. Wainwright,et al.  Embedded trees: estimation of Gaussian Processes on graphs with cycles , 2004, IEEE Transactions on Signal Processing.

[7]  Hesham El Gamal,et al.  Analyzing the turbo decoder using the Gaussian approximation , 2001, IEEE Trans. Inf. Theory.

[8]  Venkat Chandrasekaran,et al.  Estimation in Gaussian Graphical Models Using Tractable Subgraphs: A Walk-Sum Analysis , 2008, IEEE Transactions on Signal Processing.

[9]  Vijay V. Vazirani,et al.  Approximation Algorithms , 2001, Springer Berlin Heidelberg.

[10]  Sekhar Tatikonda,et al.  Loopy Belief Propogation and Gibbs Measures , 2002, UAI.

[11]  Michael Chertkov,et al.  Orbit-product representation and correction of Gaussian belief propagation , 2009, ICML '09.

[12]  Zoubin Ghahramani,et al.  A Bayesian approach to reconstructing genetic regulatory networks with hidden factors , 2005, Bioinform..

[13]  Dieter Kratsch,et al.  Feedback vertex set on AT-free graphs , 2008, Discret. Appl. Math..

[14]  Milan Sonka,et al.  Image processing analysis and machine vision [2nd ed.] , 1999 .

[15]  Iain A. Stewart,et al.  Improved upper and lower bounds on the feedback vertex numbers of grids and butterflies , 2008, Discret. Math..

[16]  T. Speed,et al.  Gaussian Markov Distributions over Finite Graphs , 1986 .

[17]  Miodrag Potkonjak,et al.  Synthesizing designs with low-cardinality minimum feedback vertex set for partial scan application , 1994, Proceedings of IEEE VLSI Test Symposium.

[18]  P. Bühlmann,et al.  Sparse graphical Gaussian modeling of the isoprenoid gene network in Arabidopsis thaliana , 2004, Genome Biology.

[19]  Béla Bollobás,et al.  Random Graphs , 1985 .

[20]  Fedor V. Fomin,et al.  On the Minimum Feedback Vertex Set Problem: Exact and Enumeration Algorithms , 2008, Algorithmica.

[21]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[22]  Mary Lou Soffa,et al.  Feedback vertex sets and cyclically reducible graphs , 1985, JACM.

[23]  Hassan A. Karimi,et al.  iGNM: a database of protein functional motions based on Gaussian Network Model , 2005, Bioinform..

[24]  Michael R. Fellows,et al.  An O(2O(k)n3) FPT Algorithm for the Undirected Feedback Vertex Set Problem , 2005, Theory of Computing Systems.

[25]  Carl Wunsch,et al.  Multiresolution optimal interpolation and statistical analysis of TOPEX/POSEIDON satellite altimetry , 1995 .

[26]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[27]  Reuven Bar-Yehuda,et al.  Approximation algorithms for the vertex feedback set problem with applications to constraint satisfaction and Bayesian inference , 1994, SODA '94.

[28]  Dmitry M. Malioutov,et al.  Approximate inference in Gaussian graphical models , 2008 .

[29]  Thomas S. Huang,et al.  Image processing , 1971 .

[30]  Richard M. Karp,et al.  Reducibility Among Combinatorial Problems , 1972, 50 Years of Integer Programming.

[31]  Danny Dolev,et al.  Fixing convergence of Gaussian belief propagation , 2009, 2009 IEEE International Symposium on Information Theory.

[32]  Carl Wunsch,et al.  Practical global oceanic state estimation , 2007 .

[33]  Adi Shamir,et al.  A Linear Time Algorithm for Finding Minimum Cutsets in Reducible Graphs , 1979, SIAM J. Comput..

[34]  Michel C. Jeruchim,et al.  Simulation of Communication Systems: Modeling, Methodology and Techniques , 2000 .

[35]  Avi Pfeffer,et al.  Loopy Belief Propagation as a Basis for Communication in Sensor Networks , 2002, UAI.

[36]  Michael I. Jordan,et al.  Loopy Belief Propagation for Approximate Inference: An Empirical Study , 1999, UAI.

[37]  Riccardo Poli,et al.  Genetic algorithm-based interactive segmentation of 3D medical images , 1999, Image Vis. Comput..

[38]  Dan Geiger,et al.  Graphical Models and Exponential Families , 1998, UAI.

[39]  Adnan Darwiche,et al.  Recursive conditioning , 2001, Artif. Intell..

[40]  Jung-Fu Cheng,et al.  Turbo Decoding as an Instance of Pearl's "Belief Propagation" Algorithm , 1998, IEEE J. Sel. Areas Commun..

[41]  Timothy A. Davis,et al.  Direct methods for sparse linear systems , 2006, Fundamentals of algorithms.

[42]  William T. Freeman,et al.  Correctness of Belief Propagation in Gaussian Graphical Models of Arbitrary Topology , 1999, Neural Computation.

[43]  Dmitry M. Malioutov,et al.  Walk-Sums and Belief Propagation in Gaussian Graphical Models , 2006, J. Mach. Learn. Res..

[44]  John W. Fisher,et al.  Loopy Belief Propagation: Convergence and Effects of Message Errors , 2005, J. Mach. Learn. Res..

[45]  Guillaume Blin,et al.  Querying Protein-Protein Interaction Networks , 2009, ISBRA.

[46]  K. Mardia.  Multi-dimensional multivariate Gaussian Markov random fields with application to image processing , 1988 .

[47]  P. Holland,et al.  An Exponential Family of Probability Distributions for Directed Graphs , 1981 .

[48]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[49]  A. George.  Nested Dissection of a Regular Finite Element Mesh , 1973 .

[50]  Judea Pearl,et al.  A Constraint-Propagation Approach to Probabilistic Reasoning , 1985, UAI.