Self-Averaging Expectation Propagation

We investigate the problem of approximate Bayesian inference for a general class of observation models by means of the expectation propagation (EP) framework for large systems under some statistical assumptions. Our approach tries to overcome the numerical bottleneck of EP caused by the inversion of large matrices. Assuming that the measurement matrices are realizations of specific types of ensembles we use the concept of freeness from random matrix theory to show that the EP cavity variances exhibit an asymptotic self-averaging property. They can be pre-computed using specific generating functions, i.e. the R- and/or S-transforms in free probability, which do not require matrix inversions. Our approach extends the framework of (generalized) approximate message passing -- assumes zero-mean iid entries of the measurement matrix -- to a general class of random matrix ensembles. The generalization is via a simple formulation of the R- and/or S-transforms of the limiting eigenvalue distribution of the Gramian of the measurement matrix. We demonstrate the performance of our approach on a signal recovery problem of nonlinear compressed sensing and compare it with that of EP.

[1]  Sundeep Rangan,et al.  Generalized approximate message passing for estimation with random linear mixing , 2010, 2011 IEEE International Symposium on Information Theory Proceedings.

[2]  S. Kirkpatrick,et al.  Solvable Model of a Spin-Glass , 1975 .

[3]  Richard G. Baraniuk,et al.  1-Bit compressive sensing , 2008, 2008 42nd Annual Conference on Information Sciences and Systems.

[4]  B. Collins,et al.  Integration with Respect to the Haar Measure on Unitary, Orthogonal and Symplectic Group , 2004, math-ph/0402073.

[5]  Yoshiyuki Kabashima,et al.  Inference from correlated patterns: a unified theory for perceptron learning and linear vector channels , 2007, ArXiv.

[6]  Toshiyuki Tanaka,et al.  A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors , 2002, IEEE Trans. Inf. Theory.

[7]  Ole Winther,et al.  A theory of solving TAP equations for Ising models with general invariant random matrices , 2015, ArXiv.

[8]  Florent Krzakala,et al.  Probabilistic reconstruction in compressed sensing: algorithms, phase diagrams, and threshold achieving matrices , 2012, ArXiv.

[9]  O. Johnson.  Free Random Variables , 2004 .

[10]  J. Sethna.  Statistical Mechanics: Entropy, Order Parameters, and Complexity , 2021 .

[11]  Brendan Farrell,et al.  Asymptotically liberating sequences of random unitary matrices , 2013, 1302.5688.

[12]  M. Mézard,et al.  Spin Glass Theory and Beyond , 1987 .

[13]  F. Guerra.  Spin Glasses , 2005, cond-mat/0507581.

[14]  Ole Winther,et al.  S-AMP for non-linear observation models , 2015, 2015 IEEE International Symposium on Information Theory (ISIT).

[15]  J. Baik,et al.  The Oxford Handbook of Random Matrix Theory , 2011 .

[16]  M. Opper,et al.  Adaptive and self-averaging Thouless-Anderson-Palmer mean-field theory for probabilistic modeling. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[17]  A. Bray,et al.  Metastable states, internal field distributions and magnetic excitations in spin glasses , 1981 .

[18]  Ralf R. Müller,et al.  Applications of Large Random Matrices in Communications Engineering , 2013, ArXiv.

[19]  Kaare Brandt Petersen,et al.  The Matrix Cookbook , 2006 .

[20]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[21]  Ole Winther,et al.  Expectation Consistent Approximate Inference , 2005, J. Mach. Learn. Res..

[22]  Ole Winther,et al.  Perturbative corrections for approximate inference in Gaussian latent variable models , 2013, J. Mach. Learn. Res..

[23]  G. Parisi,et al.  Replica field theory for deterministic models: II. A non-random spin glass with glassy behaviour , 1994, cond-mat/9406074.

[24]  Y. Kabashima.  A CDMA multiuser detection algorithm on the basis of belief propagation , 2003 .

[25]  B. Eynard,et al.  Random matrices. , 2015, 1510.04430.

[26]  Ole Winther,et al.  S-AMP: Approximate message passing for general matrix ensembles , 2014, 2014 IEEE Information Theory Workshop (ITW 2014).

[27]  Ralf R. Müller,et al.  Capacity Scaling in MIMO Systems With General Unitarily Invariant Random Matrices , 2013, IEEE Transactions on Information Theory.

[28]  Dietrich von Rosen,et al.  Cumulant-moment relation in free probability theory , 2014 .

[29]  Ole Winther,et al.  Gaussian Processes for Classification: Mean-Field Algorithms , 2000, Neural Computation.

[30]  A. Guionnet,et al.  A Fourier view on the R-transform and related asymptotics of spherical integrals , 2005 .

[31]  Tom Minka,et al.  Expectation Propagation for approximate Bayesian inference , 2001, UAI.

[32]  F. Hiai,et al.  The semicircle law, free random variables, and entropy , 2006 .

[33]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[34]  U. Haagerup,et al.  The Law of Large Numbers for the Free Multiplicative Convolution , 2012, 1211.4457.

[35]  Shlomo Shamai,et al.  Support Recovery With Sparsely Sampled Free Random Matrices , 2011, IEEE Transactions on Information Theory.

[36]  Mikko Vehkaperä,et al.  Analysis of Regularized LS Reconstruction and Random Matrix Ensembles in Compressed Sensing , 2013, IEEE Transactions on Information Theory.

[37]  M. Stephanov,et al.  Random Matrices , 2005, hep-ph/0509286.