Fast Convergent Algorithms for Expectation Propagation Approximate Bayesian Inference

We propose a novel algorithm to solve the expectation propagation relaxation of Bayesian inference for continuous-variable graphical models. In contrast to most previous algorithms, our method is provably convergent. By marrying convergent EP ideas from [15] with covariance decoupling techniques [23, 13], it runs at least an order of magnitude faster than the most common EP solver.

[1]  Heinz H. Bauschke,et al.  Fixed-Point Algorithms for Inverse Problems in Science and Engineering , 2011, Springer Optimization and Its Applications.

[2]  Matthias W. Seeger,et al.  Convex variational Bayesian inference for large scale generalized linear models , 2009, ICML '09.

[3]  Wei Chu,et al.  Gaussian Processes for Ordinal Regression , 2005, J. Mach. Learn. Res..

[4]  Matthias W. Seeger,et al.  Large Scale Bayesian Inference and Experimental Design for Sparse Linear Models , 2011, SIAM J. Imaging Sci..

[5]  Carl E. Rasmussen,et al.  Assessing Approximate Inference for Binary Gaussian Process Classification , 2005, J. Mach. Learn. Res..

[6]  Bernhard Schölkopf,et al.  Bayesian Experimental Design of Magnetic Resonance Imaging Sequences , 2008, NIPS.

[7]  Neil D. Lawrence,et al.  Semi-supervised Learning via Gaussian Processes , 2004, NIPS.

[8]  David Barber,et al.  Expectation Correction for Smoothed Inference in Switching Linear Dynamical Systems , 2006, J. Mach. Learn. Res..

[9]  Robert B. Ash,et al.  Information Theory , 2020, The SAGE International Encyclopedia of Mass Media and Society.

[10]  Florian Steinke,et al.  Bayesian Inference and Optimal Design in the Sparse Linear Model , 2007, AISTATS.

[11]  C. Rasmussen,et al.  Approximations for Binary Gaussian Process Classification , 2008 .

[12]  Ole Winther,et al.  Expectation Consistent Approximate Inference , 2005, J. Mach. Learn. Res..

[13]  Matthias W. Seeger,et al.  Gaussian Covariance and Scalable Variational Inference , 2010, ICML.

[14]  David P. Wipf,et al.  A New View of Automatic Relevance Determination , 2007, NIPS.

[15]  Tom Heskes,et al.  Efficient Bayesian multivariate fMRI analysis using a sparsifying spatio-temporal prior , 2010, NeuroImage.

[16]  L. Goddard.  Information Theory , 1962, Nature.

[17]  Tom Minka,et al.  Expectation Propagation for approximate Bayesian inference , 2001, UAI.

[18]  David S. Touretzky,et al.  Advances in neural information processing systems 2 , 1989 .

[19]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[20]  Michael I. Jordan,et al.  Sparse Gaussian Process Classification With Multiple Classes , 2004 .

[21]  Matthias Bethge,et al.  Bayesian Inference for Spiking Neuron Models with a Sparsity Prior , 2007, NIPS.

[22]  M. Opper,et al.  Adaptive and self-averaging Thouless-Anderson-Palmer mean-field theory for probabilistic modeling. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[23]  Frédo Durand,et al.  Understanding and evaluating blind deconvolution algorithms , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[25]  丸山 徹.  Convex Analysisの二,三の進展について , 1977 .

[26]  Ole Winther,et al.  Mean-Field Approaches to Independent Component Analysis , 2002, Neural Computation.

[27]  Matthias W. Seeger,et al.  Large Scale Variational Inference and Experimental Design for Sparse Generalized Linear Models , 2008, Sampling-based Optimization in the Presence of Uncertainty.