On Variational Message Passing on Factor Graphs

In this paper, it is shown how (naive and structured) variational algorithms may be derived from a factor graph by mechanically applying generic message computation rules; in this way, one can bypass error-prone variational calculus. In prior work by Bishop et al., Xing et al., and Geiger, directed and undirected graphical models have been used for this purpose. The factor graph notation amounts to simpler generic variational message computation rules; by means of factor graphs, variational methods can straightforwardly be compared to and combined with various other message-passing inference algorithms, e.g., Kalman filters and smoothers, iterated conditional modes, expectation maximization (EM), gradient methods, and particle filters. Some of those combinations have been explored in the literature, others seem to be new. Generic message computation rules for such combinations are formulated.

[1]  Hilbert J. Kappen,et al.  Mean field theory for graphical models , 2001 .

[2]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine Learning.

[3]  Justin Dauwels,et al.  Expectation maximization as message passing , 2005, Proceedings. International Symposium on Information Theory, 2005. ISIT 2005..

[4]  Wim Wiegerinck,et al.  Variational Approximations between Mean Field Theory and the Junction Tree Algorithm , 2000, UAI.

[5]  P. Stoica,et al.  Cyclic minimizers, majorization techniques, and the expectation-maximization algorithm: a refresher , 2004, IEEE Signal Process. Mag..

[6]  Christopher Meek,et al.  Structured Variational Inference Procedures and their Realizations , 2005, AISTATS.

[7]  G. Forney,et al.  Codes on graphs: normal realizations , 2000, 2000 IEEE International Symposium on Information Theory (Cat. No.00CH37060).

[8]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[9]  Andrew W. Eckford,et al.  Iterative multiuser detection with graphical modeling , 2000, 2000 IEEE International Conference on Personal Wireless Communications. Conference Proceedings (Cat. No.00TH8488).

[10]  Christian P. Robert,et al.  Monte Carlo Statistical Methods , 2005, Springer Texts in Statistics.

[11]  Justin Dauwels,et al.  On graphical models for communications and machine learning: algorithms, bounds, and analog implementation , 2006 .

[12]  Timothy J. Robinson,et al.  Sequential Monte Carlo Methods in Practice , 2003 .

[13]  Justin Dauwels,et al.  Steepest descent on factor graphs , 2005 .

[14]  Christopher M. Bishop,et al.  Structured Variational Distributions in VIBES , 2003, AISTATS.

[15]  Zoubin Ghahramani,et al.  Optimization with EM and Expectation-Conjugate-Gradient , 2003, ICML.

[16]  R. Cox,et al.  Journal of the Royal Statistical Society B , 1972 .

[17]  Matthew J. Beal,et al.  Graphical Models and Variational Methods , 2001 .

[18]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[19]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[20]  Zoubin Ghahramani,et al.  Propagation Algorithms for Variational Bayesian Learning , 2000, NIPS.

[21]  Thomas P. Minka,et al.  Divergence measures and message passing , 2005 .

[22]  Michael I. Jordan,et al.  A generalized mean field algorithm for variational inference in exponential families , 2002, UAI.

[23]  O. Seeberg.  Statistical Mechanics. — A Set of Lectures , 1975 .

[24]  Geoffrey E. Hinton,et al.  A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.

[25]  Justin Dauwels,et al.  Particle Methods as Message Passing , 2006, 2006 IEEE International Symposium on Information Theory.

[26]  Shun-ichi Amari,et al.  Methods of information geometry , 2000 .

[27]  Brendan J. Frey,et al.  A comparison of algorithms for inference and learning in probabilistic graphical models , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Michael Isard,et al.  PAMPAS: real-valued graphical models for computer vision , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[29]  Michael I. Jordan,et al.  Graph Partition Strategies for Generalized Mean Field Inference , 2004, UAI.

[30]  Michael I. Jordan,et al.  Bayesian parameter estimation via variational methods , 2000, Stat. Comput..

[31]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[32]  G. Parisi,et al.  Statistical Field Theory , 1988 .

[33]  G.D. Forney,et al.  Codes on graphs: Normal realizations , 2000, IEEE Trans. Inf. Theory.

[34]  Brendan J. Frey,et al.  Factor graphs and the sum-product algorithm , 2001, IEEE Trans. Inf. Theory.

[35]  H.-A. Loeliger,et al.  An introduction to factor graphs , 2004, IEEE Signal Process. Mag..

[36]  Kaare Brandt Petersen,et al.  On the Slow Convergence of EM and VBEM in Low-Noise Linear Models , 2005, Neural Computation.

[37]  Subbarayan Pasupathy,et al.  Reduced-complexity turbo receivers for single and multi-antenna systems via variational inference in factor graphs , 2004, 2004 IEEE International Conference on Communications (IEEE Cat. No.04CH37577).

[38]  A. Doucet,et al.  Sequential auxiliary particle belief propagation , 2005, 2005 7th International Conference on Information Fusion.

[39]  James C. Bezdek,et al.  Some Notes on Alternating Optimization , 2002, AFSS.

[40]  John M. Winn,et al.  Variational Message Passing and its Applications , 2004 .

[41]  Manfred Opper,et al.  Graphical Models and Variational Methods , 2001 .

[42]  Matthew J. Beal.  Variational algorithms for approximate Bayesian inference , 2003 .

[43]  Charles M. Bishop,et al.  Variational Message Passing , 2005, J. Mach. Learn. Res..

[44]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[45]  Zoubin Ghahramani,et al.  The variational Kalman smoother , 2001 .

[46]  G. McLachlan,et al.  The EM algorithm and extensions , 1996 .

[47]  Nando de Freitas,et al.  Variational MCMC , 2001, UAI.