The dynamics of message passing on dense graphs, with applications to compressed sensing

“Approximate message passing” (AMP) algorithms have proved to be effective in reconstructing sparse signals from a small number of incoherent linear measurements. Extensive numerical experiments further showed that their dynamics is accurately tracked by a simple one-dimensional iteration termed state evolution. In this paper, we provide rigorous foundation to state evolution. We prove that indeed it holds asymptotically in the large system limit for sensing matrices with independent and identically distributed Gaussian entries. While our focus is on message passing algorithms for compressed sensing, the analysis extends beyond this setting, to a general class of algorithms on dense graphs. In this context, state evolution plays the role that density evolution has for sparse graphs. The proof technique is fundamentally different from the standard approach to density evolution, in that it copes with a large number of short cycles in the underlying factor graph. It relies instead on a conditioning technique recently developed by Erwin Bolthausen in the context of spin glass theory.

[1]  I. Johnstone,et al.  Minimax risk overlp-balls forlp-error , 1994 .

[2]  M. Mézard,et al.  Information, Physics, and Computation , 2009 .

[3]  Devavrat Shah,et al.  Belief Propagation: An Asymptotically Optimal Algorithm for the Random Assignment Problem , 2009, Math. Oper. Res..

[4]  C. Stein A bound for the error in the normal approximation to the distribution of a sum of dependent random variables , 1972 .

[5]  D. Donoho,et al.  Counting faces of randomly-projected polytopes when the projection radically lowers dimension , 2006, math/0607364.

[6]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[7]  Yoshiyuki Kabashima,et al.  Erratum: A typical reconstruction limit of compressed sensing based on Lp-norm minimization , 2009, ArXiv.

[8]  Andrea Montanari,et al.  The dynamics of message passing on dense graphs, with applications to compressed sensing , 2010, ISIT.

[9]  Dongning Guo,et al.  Asymptotic Mean-Square Optimality of Belief Propagation for Sparse Linear Systems , 2006, 2006 IEEE Information Theory Workshop - ITW '06 Chengdu.

[10]  David Saad,et al.  Improved message passing for inference in densely connected systems , 2005, ArXiv.

[11]  David Tse,et al.  Linear Multiuser Receivers: Effective Interference, Effective Bandwidth and User Capacity , 1999, IEEE Trans. Inf. Theory.

[12]  Tim Austin Mean field models for spin glasses , 2012 .

[13]  Johan Wästlund AN EASY PROOF OF THE ζ ( 2 ) LIMIT IN THE RANDOM ASSIGNMENT PROBLEM , 2006 .

[14]  D. Donoho,et al.  Neighborliness of randomly projected simplices in high dimensions. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[15]  I. Benjamini,et al.  Percolation Beyond $Z^d$, Many Questions And a Few Answers , 1996 .

[16]  I. Johnstone,et al.  Minimax Risk over l p-Balls for l q-error , 1994 .

[17]  Andrea Montanari,et al.  Analysis of Belief Propagation for Non-Linear Problems: The Example of CDMA (or: How to Prove Tanaka's Formula) , 2006, 2006 IEEE Information Theory Workshop - ITW '06 Punta del Este.

[18]  David Aldous,et al.  The ζ(2) limit in the random assignment problem , 2000, Random Struct. Algorithms.

[19]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[20]  Andrea Montanari,et al.  The LASSO Risk for Gaussian Matrices , 2010, IEEE Transactions on Information Theory.

[21]  Shlomo Shamai,et al.  Spectral Efficiency of CDMA with Random Spreading , 1999, IEEE Trans. Inf. Theory.

[22]  V. Marčenko,et al.  DISTRIBUTION OF EIGENVALUES FOR SOME SETS OF RANDOM MATRICES , 1967 .

[23]  Masato Okada,et al.  Approximate belief propagation, density evolution, and statistical neurodynamics for CDMA multiuser detection , 2005, IEEE Transactions on Information Theory.

[24]  Andrea Montanari,et al.  The Noise-Sensitivity Phase Transition in Compressed Sensing , 2010, IEEE Transactions on Information Theory.

[25]  M. Talagrand Mean Field Models for Spin Glasses , 2011 .

[26]  Sundeep Rangan,et al.  Asymptotic Analysis of MAP Estimation via the Replica Method and Applications to Compressed Sensing , 2009, IEEE Transactions on Information Theory.

[27]  Toshiyuki Tanaka,et al.  A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors , 2002, IEEE Trans. Inf. Theory.

[28]  Y. Kabashima A CDMA multiuser detection algorithm on the basis of belief propagation , 2003 .

[29]  D. L. Donoho,et al.  Ideal spacial adaptation via wavelet shrinkage , 1994 .

[30]  T. Hu,et al.  On the strong law for arrays and for the bootstrap mean and variance , 1997 .

[31]  Toshiyuki TANAKA Statistical Physics of Information Processing in Mobile Communications , 2005 .

[32]  J. Michael Steele,et al.  The Objective Method: Probabilistic Combinatorial Optimization and Local Weak Convergence , 2004 .

[33]  Rüdiger L. Urbanke,et al.  Modern Coding Theory , 2008 .

[34]  Andrea Montanari,et al.  Message passing algorithms for compressed sensing: I. motivation and construction , 2009, 2010 IEEE Information Theory Workshop on Information Theory (ITW 2010, Cairo).

[35]  Dongning Guo,et al.  A single-letter characterization of optimal noisy compressed sensing , 2009, 2009 47th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[36]  M. Talagrand On the High Temperature Phase of the Sherrington-Kirkpatrick Model , 2002 .

[37]  Arian Maleki,et al.  Optimally Tuned Iterative Reconstruction Algorithms for Compressed Sensing , 2009, IEEE Journal of Selected Topics in Signal Processing.

[38]  Sergio Verdú,et al.  Randomly spread CDMA: asymptotics via statistical physics , 2005, IEEE Transactions on Information Theory.

[39]  D. Donoho For most large underdetermined systems of equations, the minimal 𝓁1‐norm near‐solution approximates the sparsest near‐solution , 2006 .