Understanding and improving belief propagation

The research reported in this thesis focuses on approximation techniques for inference in graphical models. Approximate inference can be dTefined as the task of calculating approximations for certain probabilities in large, complexprobabilistic models, such as Bayesian networks, Markov random fields or Ising spin systems (which are all special cases of graphical models). We have focussed on graphical models with variables that have a finite number of possible values. Calculating these probabilities is simple in principle, but computationally hard in practice, because it requires a summation over an exponential number of terms. However, the practical relevance is enormous: application areas include genetic linkage analysis, medical diagnosis, expert systems, error correcting codes, speech recognition, computer vision and many more. Because the probabilities that one is interested in cannot always be calculated exactly (given a limited amount of computation time), one often uses approximate methods, which use less computation time but only give an approximation of the quantities of interest. In this thesis, we have tried to better understand and improve upon Belief Propagation (BP), a popular approximate inference method that performs surprisingly well on many problems. It is also known as 'Loopy Belief Propagation', the 'Sum-Product Algorithm' and the 'Bethe-Peierls approximation'. BP yields exact results if the underlying graphical model has no loops. In general, the BP results are approximate but can be surprisingly accurate. However, if variables become highly dependent, the error made by BP can become significant. In some cases, BP does not even converge anymore. Our results have contributed to a better understanding of these issues. In addition, we introduced a method that improves the accuracy of BP by taking into account the influence of loops in the graphical model. Finally, we proposed a method to calculate exact bounds on marginal probabilities, which was inspired by BP

[1]  Robert G. Gallager,et al.  Low-density parity-check codes , 1962, IRE Trans. Inf. Theory.

[2]  W. Wiegerinck,et al.  Approximate inference techniques with expectation constraints , 2005 .

[3]  Yee Whye Teh,et al.  Belief Optimization for Binary Networks: A Stable Alternative to Loopy Belief Propagation , 2001, UAI.

[4]  Riccardo Zecchina,et al.  Survey propagation: An algorithm for satisfiability , 2002, Random Struct. Algorithms.

[5]  H. Kappen,et al.  Spin-glass phase transitions on real-world graphs , 2004, cond-mat/0408378.

[6]  John W. Fisher,et al.  Message Errors in Belief Propagation , 2004, NIPS.

[7]  Alan Brunton,et al.  Belief Propagation for Panorama Generation , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[8]  Hilbert J. Kappen,et al.  Bound Propagation , 2003, J. Artif. Intell. Res..

[9]  R. Zecchina,et al.  Ferromagnetic ordering in graphs with arbitrary degree distribution , 2002, cond-mat/0203416.

[10]  Rina Dechter,et al.  Iterative Join-Graph Propagation , 2002, UAI.

[11]  Kazuyuki Tanaka.  Statistical-mechanical approach to image processing , 2002 .

[12]  Dahlia Malkhi,et al.  Efficient Large Scale Content Distribution , 2004 .

[13]  Alan L. Yuille,et al.  CCCP Algorithms to Minimize the Bethe and Kikuchi Free Energies: Convergent Alternatives to Belief Propagation , 2002, Neural Computation.

[14]  Yuan Qi,et al.  Tree-structured Approximations by Expectation Propagation , 2003, NIPS.

[15]  L. YuilleA..  CCCP algorithms to minimize the Bethe and Kikuchi free energies , 2002 .

[16]  Jung-Fu Cheng,et al.  Turbo Decoding as an Instance of Pearl's "Belief Propagation" Algorithm , 1998, IEEE J. Sel. Areas Commun..

[17]  Tom Heskes,et al.  On the Uniqueness of Loopy Belief Propagation Fixed Points , 2004, Neural Computation.

[18]  Josef Stoer,et al.  Numerische Mathematik 1 , 1989 .

[19]  Tom Heskes,et al.  Fractional Belief Propagation , 2002, NIPS.

[20]  A. Scott,et al.  The Repulsive Lattice Gas, the Independent-Set Polynomial, and the Lovász Local Lemma , 2003, cond-mat/0309352.

[21]  Ian McGraw,et al.  Residual Belief Propagation: Informed Scheduling for Asynchronous Message Passing , 2006, UAI.

[22]  Michael Chertkov,et al.  Loop series for discrete statistical models on graphs , 2006, ArXiv.

[23]  Huicheng Zheng,et al.  From Maximum Entropy to Belief Propagation: An application to Skin Detection , 2004, BMVC.

[24]  John W. Fisher,et al.  Loopy Belief Propagation: Convergence and Effects of Message Errors , 2005, J. Mach. Learn. Res..

[25]  R. Kikuchi.  A Theory of Cooperative Phenomena , 1951 .

[26]  D. Heckerman,et al.  ,81. Introduction , 2022 .

[27]  M. Mézard,et al.  The Bethe lattice spin glass revisited , 2000, cond-mat/0009418.

[28]  Bruce D'Ambrosio,et al.  Multiplicative Factorization of Noisy-Max , 1999, UAI.

[29]  Hilbert J. Kappen,et al.  Validity Estimates for Loopy Belief Propagation on Binary Real-world Networks , 2004, NIPS.

[30]  M. Mézard,et al.  Spin Glass Theory and Beyond , 1987 .

[31]  Michael I. Mandel,et al.  Visual Hand Tracking Using Nonparametric Belief Propagation , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[32]  W. Freeman,et al.  Generalized Belief Propagation , 2000, NIPS.

[33]  Brendan J. Frey,et al.  A Revolution: Belief Propagation in Graphs with Cycles , 1997, NIPS.

[34]  James M. Coughlan,et al.  Finding Deformable Shapes Using Loopy Belief Propagation , 2002, ECCV.

[35]  Carl D. Meyer,et al.  Matrix Analysis and Applied Linear Algebra , 2000 .

[36]  William T. Freeman,et al.  Nonparametric belief propagation , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[37]  Michael I. Jordan,et al.  Variational Probabilistic Inference and the QMR-DT Network , 2011, J. Artif. Intell. Res..

[38]  Hans Maassen,et al.  Quantitative imaging through a spectrograph. 1. Principles and theory. , 2004, Applied optics.

[39]  Michael I. Jordan,et al.  Recursive Algorithms for Approximating Probabilities in Graphical Models , 1996, NIPS.

[40]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[41]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[42]  Y. Kabashima.  Propagating beliefs in spin-glass models , 2002, cond-mat/0211500.

[43]  William T. Freeman,et al.  On the optimality of solutions of the max-product belief-propagation algorithm in arbitrary graphs , 2001, IEEE Trans. Inf. Theory.

[44]  Riccardo Zecchina,et al.  Survey propagation as local equilibrium equations , 2003, ArXiv.

[45]  R. Baxter.  Exactly solved models in statistical mechanics , 1982 .

[46]  Gregory F. Cooper,et al.  The Computational Complexity of Probabilistic Inference Using Bayesian Belief Networks , 1990, Artif. Intell..

[47]  Dror Weitz,et al.  Counting independent sets up to the tree threshold , 2006, STOC '06.

[48]  Daniel P. Huttenlocher,et al.  Efficient Belief Propagation for Early Vision , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[49]  K. Nakanishi.  Two- and three-spin cluster theory of spin-glasses , 1981 .

[50]  J. Pearl.  Causality: Models, Reasoning and Inference , 2000 .

[51]  Nanning Zheng,et al.  Stereo Matching Using Belief Propagation , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[52]  James M. Coughlan,et al.  Shape Matching with Belief Propagation: Using Dynamic Quantization to Accomodate Occlusion and Clutter , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[53]  David J. Spiegelhalter,et al.  Local computations with probabilities on graphical structures and their application to expert systems , 1990 .

[54]  Alessandro Pelizzola,et al.  Cluster Variation Method in Statistical Physics and Probabilistic Graphical Models , 2005, ArXiv.

[55]  Thomas S. Huang,et al.  Non-parametric image super-resolution using multiple images , 2005, IEEE International Conference on Image Processing 2005.

[56]  Ole Winther,et al.  Expectation Consistent Approximate Inference , 2005, J. Mach. Learn. Res..

[57]  Tom Minka,et al.  Expectation Propagation for approximate Bayesian inference , 2001, UAI.

[58]  J. Calvin Giddings,et al.  Principles and theory , 1965 .

[59]  Emeric Deutsch.  On matrix norms and logarithmic norms , 1975 .

[60]  M. F.,et al.  Bibliography , 1985, Experimental Gerontology.

[61]  B Wemmenhove,et al.  Replica symmetry breaking in the 'small world' spin glass , 2005 .

[62]  Yair Weiss,et al.  Correctness of Local Probability Propagation in Graphical Models with Loops , 2000, Neural Computation.

[63]  Nobuyuki Taga,et al.  On the Convergence of Loopy Belief Propagation Algorithm for Different Update Rules , 2006, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[64]  M. Newman,et al.  Random graphs with arbitrary degree distributions and their applications. , 2000, Physical review. E, Statistical, nonlinear, and soft matter physics.

[65]  William T. Freeman,et al.  Example-Based Super-Resolution , 2002, IEEE Computer Graphics and Applications.

[66]  Brendan J. Frey,et al.  Enforcing integrability for surface reconstruction algorithms using belief propagation in graphical models , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[67]  William T. Freeman,et al.  Comparison of graph cuts with belief propagation for stereo, using identical MRF parameters , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[68]  Engelbert Hubbers,et al.  A polynomial counterexample to the Markus-Yamabe conjecture , 1997 .

[69]  Y Mao,et al.  Lifetime probability of developing lung cancer, by smoking status, Canada. , 1994, Canadian journal of public health = Revue canadienne de sante publique.

[70]  Avi Pfeffer,et al.  Loopy Belief Propagation as a Basis for Communication in Sensor Networks , 2002, UAI.

[71]  Dan Roth,et al.  On the Hardness of Approximate Reasoning , 1993, IJCAI.

[72]  William T. Freeman,et al.  Learning Low-Level Vision , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[73]  Michael I. Jordan,et al.  Loopy Belief Propagation for Approximate Inference: An Empirical Study , 1999, UAI.

[74]  Jianbo Shi,et al.  Multiple frame motion inference using belief propagation , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[75]  D. Thouless,et al.  Stability of the Sherrington-Kirkpatrick solution of a spin glass model , 1978 .

[76]  Yee Whye Teh,et al.  Structured Region Graphs: Morphing EP into GBP , 2005, UAI.

[77]  Brendan J. Frey,et al.  Factor graphs and the sum-product algorithm , 2001, IEEE Trans. Inf. Theory.

[78]  Thomas G. Dietterich,et al.  Editors. Advances in Neural Information Processing Systems , 2002 .

[79]  Carlo Zaniolo,et al.  Data cleaning using belief propagation , 2005, IQIS '05.

[80]  Hilbert J. Kappen,et al.  Sufficient Conditions for Convergence of Loopy Belief Propagation , 2005, UAI.

[81]  Hilbert J. Kappen,et al.  Approximate inference for medical diagnosis , 1999, Pattern Recognit. Lett..

[82]  Hilbert J. Kappen,et al.  Loop corrections for approximate inference , 2006, ArXiv.

[83]  Michael Chertkov,et al.  Loop Calculus Helps to Improve Belief Propagation and Linear Programming Decodings of Low-Density-Parity-Check Codes , 2006, ArXiv.

[84]  H. Bethe.  Statistical Theory of Superlattices , 1935 .

[85]  Michael Luby,et al.  Approximating Probabilistic Inference in Bayesian Belief Networks is NP-Hard , 1993, Artif. Intell..

[86]  Nobuyuki Taga,et al.  Error Bounds Between Marginal Probabilities and Beliefs of Loopy Belief Propagation Algorithm , 2006, MICAI.

[87]  Hilbert J. Kappen,et al.  A Tighter Bound for Graphical Models , 2001, Neural Computation.

[88]  Tom Heskes,et al.  Convexity Arguments for Efficient Minimization of the Bethe and Kikuchi Free Energies , 2006, J. Artif. Intell. Res..

[89]  A. Montanari,et al.  How to compute loop corrections to the Bethe approximation , 2005, cond-mat/0506769.

[90]  G. Parisi,et al.  Loop expansion around the Bethe–Peierls approximation for lattice models , 2005, cond-mat/0512529.

[91]  Sekhar C. Tatikonda,et al.  Convergence of the sum-product algorithm , 2003, Proceedings 2003 IEEE Information Theory Workshop (Cat. No.03EX674).

[92]  Brendan J. Frey,et al.  Very loopy belief propagation for unwrapping phase images , 2001, NIPS.

[93]  Hans-Otto Georgii,et al.  Gibbs Measures and Phase Transitions , 1988 .

[94]  Hilbert J. Kappen,et al.  Approximate Inference and Constrained Optimization , 2002, UAI.

[95]  西森 秀稔.  Statistical physics of spin glasses and information processing : an introduction , 2001 .

[96]  Hilbert J. Kappen,et al.  On the properties of the Bethe approximation and loopy belief propagation on binary networks , 2004 .

[97]  Martin J. Wainwright,et al.  A new class of upper bounds on the log partition function , 2002, IEEE Transactions on Information Theory.

[98]  John W. Fisher,et al.  Nonparametric belief propagation for self-localization of sensor networks , 2005, IEEE Journal on Selected Areas in Communications.

[99]  Sekhar Tatikonda,et al.  Loopy Belief Propogation and Gibbs Measures , 2002, UAI.