S-AMP: Approximate message passing for general matrix ensembles

We propose a novel iterative estimation algorithm for linear observation models called S-AMP. The fixed points of S-AMP are the stationary points of the exact Gibbs free energy under a set of (first- and second-) moment consistency constraints in the large system limit. S-AMP extends the approximate message-passing (AMP) algorithm to general matrix ensembles with a well-defined large system size limit. The generalization is based on the S-transform (in free probability) of the spectrum of the measurement matrix. Furthermore, we show that the optimality of S-AMP follows directly from its design rather than from solving a separate optimization problem as done for AMP.

[1]  Ole Winther,et al.  Expectation Consistent Approximate Inference , 2005, J. Mach. Learn. Res..

[2]  M. Opper,et al.  From Naive Mean Field Theory to the TAP Equations , 2001 .

[3]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[4]  Mikko Vehkaperä,et al.  Analysis of Regularized LS Reconstruction and Random Matrix Ensembles in Compressed Sensing , 2013, IEEE Transactions on Information Theory.

[5]  Sundeep Rangan,et al.  Generalized approximate message passing for estimation with random linear mixing , 2010, 2011 IEEE International Symposium on Information Theory Proceedings.

[6]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[7]  T. Heskes,et al.  Expectation propagation for approximate inference in dynamic bayesian networks , 2002, UAI 2002.

[8]  M. Opper,et al.  Adaptive and self-averaging Thouless-Anderson-Palmer mean-field theory for probabilistic modeling. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[9]  Antonia Maria Tulino,et al.  Random Matrix Theory and Wireless Communications , 2004, Found. Trends Commun. Inf. Theory.

[10]  Florent Krzakala,et al.  On convergence of approximate message passing , 2014, 2014 IEEE International Symposium on Information Theory.

[11]  W. Wiegerinck,et al.  Approximate inference techniques with expectation constraints , 2005 .

[12]  Andrea Montanari,et al.  Message passing algorithms for compressed sensing: I. motivation and construction , 2009, 2010 IEEE Information Theory Workshop on Information Theory (ITW 2010, Cairo).

[13]  Florian Steinke,et al.  Bayesian Inference and Optimal Design in the Sparse Linear Model , 2007, AISTATS.

[14]  Florent Krzakala,et al.  Probabilistic reconstruction in compressed sensing: algorithms, phase diagrams, and threshold achieving matrices , 2012, ArXiv.

[15]  Ralf R. Müller,et al.  Applications of Large Random Matrices in Communications Engineering , 2013, ArXiv.

[16]  Mikko Vehkaperä,et al.  Signal recovery using expectation consistent approximation for linear observations , 2014, 2014 IEEE International Symposium on Information Theory.

[17]  Andrea Montanari,et al.  The dynamics of message passing on dense graphs, with applications to compressed sensing , 2010, ISIT.

[18]  Erwin Riegler,et al.  Merging belief propagation and the mean field approximation: A free energy approach , 2010 .

[19]  Mathieu S. Capcarrère,et al.  Necessary conditions for density classification by cellular automata. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.