The dynamics of message passing on dense graphs, with applications to compressed sensing

‘Approximate message passing’ algorithms proved to be extremely effective in reconstructing sparse signals from a small number of incoherent linear measurements. Extensive numerical experiments further showed that their dynamics is accurately tracked by a simple one-dimensional iteration termed state evolution. In this paper we provide the first rigorous foundation to state evolution. We prove that indeed it holds asymptotically in the large system limit for sensing matrices with iid gaussian entries. While our focus is on message passing algorithms for compressed sensing, the analysis extends beyond this setting, to a general class of algorithms on dense graphs. In this context, state evolution plays the role that density evolution has for sparse graphs.

[1]  V. Marčenko,et al.  DISTRIBUTION OF EIGENVALUES FOR SOME SETS OF RANDOM MATRICES , 1967 .

[2]  C. Stein.  A bound for the error in the normal approximation to the distribution of a sum of dependent random variables , 1972 .

[3]  I. Johnstone,et al.  Minimax risk overlp-balls forlp-error , 1994 .

[4]  I. Benjamini,et al.  Percolation Beyond $Z^d$, Many Questions And a Few Answers , 1996 .

[5]  T. Hu,et al.  On the strong law for arrays and for the bootstrap mean and variance , 1997 .

[6]  David Tse,et al.  Linear Multiuser Receivers: Effective Interference, Effective Bandwidth and User Capacity , 1999, IEEE Trans. Inf. Theory.

[7]  Shlomo Shamai,et al.  Spectral Efficiency of CDMA with Random Spreading , 1999, IEEE Trans. Inf. Theory.

[8]  D. Aldous.  The ζ(2) limit in the random assignment problem , 2000, Random Struct. Algorithms.

[9]  Toshiyuki Tanaka,et al.  A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors , 2002, IEEE Trans. Inf. Theory.

[10]  M. Talagrand.  On the High Temperature Phase of the Sherrington-Kirkpatrick Model , 2002 .

[11]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[12]  Y. Kabashima.  A CDMA multiuser detection algorithm on the basis of belief propagation , 2003 .

[13]  J. Michael Steele,et al.  The Objective Method: Probabilistic Combinatorial Optimization and Local Weak Convergence , 2004 .

[14]  Sergio Verdú,et al.  Randomly spread CDMA: asymptotics via statistical physics , 2005, IEEE Transactions on Information Theory.

[15]  Toshiyuki TANAKA.  Statistical Physics of Information Processing in Mobile Communications , 2005 .

[16]  Masato Okada,et al.  Approximate belief propagation, density evolution, and statistical neurodynamics for CDMA multiuser detection , 2005, IEEE Transactions on Information Theory.

[17]  David Saad,et al.  Improved message passing for inference in densely connected systems , 2005, ArXiv.

[18]  D. Donoho,et al.  Neighborliness of randomly projected simplices in high dimensions. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[19]  D. Donoho,et al.  Counting faces of randomly-projected polytopes when the projection radically lowers dimension , 2006, math/0607364.

[20]  D. Donoho.  For most large underdetermined systems of equations, the minimal 𝓁1‐norm near‐solution approximates the sparsest near‐solution , 2006 .

[21]  Dongning Guo,et al.  Asymptotic Mean-Square Optimality of Belief Propagation for Sparse Linear Systems , 2006, 2006 IEEE Information Theory Workshop - ITW '06 Chengdu.

[22]  Andrea Montanari,et al.  Analysis of Belief Propagation for Non-Linear Problems: The Example of CDMA (or: How to Prove Tanaka's Formula) , 2006, 2006 IEEE Information Theory Workshop - ITW '06 Punta del Este.

[23]  Rüdiger L. Urbanke,et al.  Modern Coding Theory , 2008 .

[24]  Sundeep Rangan,et al.  Asymptotic Analysis of MAP Estimation via the Replica Method and Compressed Sensing , 2009, NIPS.

[25]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[26]  Yoshiyuki Kabashima,et al.  Erratum: A typical reconstruction limit of compressed sensing based on Lp-norm minimization , 2009, ArXiv.

[27]  M. Mézard,et al.  Information, Physics, and Computation , 2009 .

[28]  Dongning Guo,et al.  A single-letter characterization of optimal noisy compressed sensing , 2009, 2009 47th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[29]  Andrea Montanari,et al.  Message passing algorithms for compressed sensing: I. motivation and construction , 2009, 2010 IEEE Information Theory Workshop on Information Theory (ITW 2010, Cairo).

[30]  Andrea Montanari,et al.  The Noise-Sensitivity Phase Transition in Compressed Sensing , 2010, IEEE Transactions on Information Theory.

[31]  M. Talagrand.  Mean Field Models for Spin Glasses , 2011 .

[32]  Andrea Montanari,et al.  The LASSO Risk for Gaussian Matrices , 2010, IEEE Transactions on Information Theory.

[33]  Sundeep Rangan,et al.  Asymptotic Analysis of MAP Estimation via the Replica Method and Applications to Compressed Sensing , 2009, IEEE Transactions on Information Theory.