Merging belief propagation and the mean field approximation: A free energy approach

We present a joint message passing approach that combines belief propagation and the mean field approximation. Our analysis is based on the region-based free energy approximation method proposed by Yedidia et al., which allows to use the same objective function (Kullback-Leibler divergence) as a starting point. In this method message passing fixed point equations (which correspond to the update rules in a message passing algorithm) are then obtained by imposing different region-based approximations and constraints on the mean field and belief propagation parts of the corresponding factor graph. Our results can be applied, for example, to algorithms that perform joint channel estimation and decoding in iterative receivers. This is demonstrated in a simple example.

[1]  Shun-ichi Amari,et al.  Information geometry of turbo and low-density parity-check codes , 2004, IEEE Transactions on Information Theory.

[2]  B. D. Craven,et al.  A generalization of Lagrange multipliers , 1970, Bulletin of the Australian Mathematical Society.

[3]  Solomon Kullback,et al.  Information Theory and Statistics , 1960 .

[4]  Justin Dauwels,et al.  On Variational Message Passing on Factor Graphs , 2007, 2007 IEEE International Symposium on Information Theory.

[5]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[6]  Justin Dauwels,et al.  Expectation maximization as message passing , 2005, Proceedings. International Symposium on Information Theory, 2005. ISIT 2005..

[7]  Bin Hu.  A Variational Bayesian Framework Divergence Minimization and Its Application in CDMA Receivers , 2010 .

[8]  H. Vincent Poor,et al.  Iterative (turbo) soft interference cancellation and decoding for coded CDMA , 1999, IEEE Trans. Commun..

[9]  Shun-ichi Amari,et al.  Methods of information geometry , 2000 .

[10]  Henk Wymeersch,et al.  Uniformly reweighted belief propagation: A factor graph approach , 2011, 2011 IEEE International Symposium on Information Theory Proceedings.

[11]  Charles M. Bishop,et al.  Variational Message Passing , 2005, J. Mach. Learn. Res..

[12]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[13]  Erwin Riegler,et al.  Receiver Architectures for MIMO-OFDM Based on a Combined VMP-SP Algorithm , 2011, ArXiv.

[14]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine Learning.

[15]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[16]  H. Bethe.  Statistical Theory of Superlattices , 1935 .

[17]  Thomas P. Minka,et al.  Divergence measures and message passing , 2005 .

[18]  Gunther Auer,et al.  Graph-based soft iterative receiver for higher-order modulation , 2010, 2010 IEEE 12th International Conference on Communication Technology.

[19]  Li Ping,et al.  The Factor Graph Approach to Model-Based Signal Processing , 2007, Proceedings of the IEEE.

[20]  Toshiyuki Tanaka,et al.  Information Geometry of Mean-Field Approximation , 2000, Neural Computation.

[21]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[22]  T. Aaron Gulliver,et al.  Cross-Entropy and Iterative Decoding , 1998, IEEE Trans. Inf. Theory.

[23]  A.W. Eckford.  The factor graph EM Algorithm: applications for LDPC codes , 2005, IEEE 6th Workshop on Signal Processing Advances in Wireless Communications, 2005..

[24]  Brendan J. Frey,et al.  Factor graphs and the sum-product algorithm , 2001, IEEE Trans. Inf. Theory.

[25]  Moe Z. Win,et al.  Cooperative Localization in Wireless Networks , 2009, Proceedings of the IEEE.

[26]  Troels Pedersen,et al.  A variational message passing algorithm for sensor self-localization in wireless networks , 2011, 2011 IEEE International Symposium on Information Theory Proceedings.

[27]  C. Richard Johnson,et al.  A refined information geometric interpretation of turbo decoding , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[28]  G. Parisi,et al.  Statistical Field Theory , 1988 .

[29]  Tom Heskes,et al.  Stable Fixed Points of Loopy Belief Propagation Are Local Minima of the Bethe Free Energy , 2002, NIPS.

[30]  Erwin Riegler,et al.  Variational Message-Passing for Joint Channel Estimation and Decoding in MIMO-OFDM , 2010, 2010 IEEE Global Telecommunications Conference GLOBECOM 2010.

[31]  Christoph F. Mecklenbräuker,et al.  Iterative CDMA Multiuser Receiver With Soft Decision-Directed Channel Estimation , 2006, IEEE Transactions on Signal Processing.

[32]  D. Luenberger.  Optimization by Vector Space Methods , 1968 .

[33]  Martin J. Wainwright,et al.  A new class of upper bounds on the log partition function , 2002, IEEE Transactions on Information Theory.

[34]  Ralf R. Müller,et al.  Joint Twofold-Iterative Channel Estimation and Multiuser Detection for MIMO-OFDM Systems , 2008, IEEE Transactions on Wireless Communications.