State Evolution for Approximate Message Passing with Non-Separable Functions

Given a high-dimensional data matrix ${\boldsymbol A}\in{\mathbb R}^{m\times n}$, Approximate Message Passing (AMP) algorithms construct sequences of vectors ${\boldsymbol u}^t\in{\mathbb R}^n$, ${\boldsymbol v}^t\in{\mathbb R}^m$, indexed by $t\in\{0,1,2\dots\}$ by iteratively applying ${\boldsymbol A}$ or ${\boldsymbol A}^{\sf T}$, and suitable non-linear functions, which depend on the specific application. Special instances of this approach have been developed --among other applications-- for compressed sensing reconstruction, robust regression, Bayesian estimation, low-rank matrix recovery, phase retrieval, and community detection in graphs. For certain classes of random matrices ${\boldsymbol A}$, AMP admits an asymptotically exact description in the high-dimensional limit $m,n\to\infty$, which goes under the name of `state evolution.' Earlier work established state evolution for separable non-linearities (under certain regularity conditions). Nevertheless, empirical work demonstrated several important applications that require non-separable functions. In this paper we generalize state evolution to Lipschitz continuous non-separable nonlinearities, for Gaussian matrices ${\boldsymbol A}$. Our proof makes use of Bolthausen's conditioning technique along with several approximation arguments. In particular, we introduce a modified algorithm (called LAMP for Long AMP) which is of independent interest.

[1]  Jean-Michel Morel,et al.  A Review of Image Denoising Algorithms, with a New One , 2005, Multiscale Model. Simul..

[2]  Philip Schniter,et al.  Turbo reconstruction of structured sparse signals , 2010, 2010 44th Annual Conference on Information Sciences and Systems (CISS).

[3]  Florent Krzakala,et al.  Phase Transitions and Sample Complexity in Bayes-Optimal Matrix Factorization , 2014, IEEE Transactions on Information Theory.

[4]  Gábor Lugosi,et al.  Concentration Inequalities - A Nonasymptotic Theory of Independence , 2013, Concentration Inequalities.

[5]  C. Stein A bound for the error in the normal approximation to the distribution of a sum of dependent random variables , 1972 .

[6]  Pablo A. Parrilo,et al.  The Convex Geometry of Linear Inverse Problems , 2010, Foundations of Computational Mathematics.

[7]  Christos Thrampoulidis,et al.  The squared-error of generalized LASSO: A precise analysis , 2013, 2013 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[8]  Andrea Montanari,et al.  The dynamics of message passing on dense graphs, with applications to compressed sensing , 2010, ISIT.

[9]  R. Palmer,et al.  Solution of 'Solvable model of a spin glass' , 1977 .

[10]  Adel Javanmard,et al.  State Evolution for General Approximate Message Passing Algorithms, with Applications to Spatial Coupling , 2012, ArXiv.

[11]  Ramji Venkataramanan,et al.  Finite-sample analysis of Approximate Message Passing , 2016, 2016 IEEE International Symposium on Information Theory (ISIT).

[12]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[13]  Andrea Montanari,et al.  Accurate Prediction of Phase Transitions in Compressed Sensing via a Connection to Minimax Denoising , 2011, IEEE Transactions on Information Theory.

[14]  D. Donoho,et al.  Minimax risk of matrix denoising by singular value thresholding , 2013, 1304.2085.

[15]  C. Givens,et al.  A class of Wasserstein metrics for probability distributions. , 1984 .

[16]  Andrea Montanari,et al.  The Noise-Sensitivity Phase Transition in Compressed Sensing , 2010, IEEE Transactions on Information Theory.

[17]  Sundeep Rangan,et al.  Hybrid Approximate Message Passing , 2011, IEEE Transactions on Signal Processing.

[18]  Li Ping,et al.  Orthogonal AMP , 2016, IEEE Access.

[19]  I. Prucha,et al.  Central Limit Theorems and Uniform Laws of Large Numbers for Arrays of Random Fields. , 2009, Journal of econometrics.

[20]  Andrea Montanari,et al.  The phase transition of matrix recovery from Gaussian measurements matches the minimax MSE of matrix denoising , 2013, Proceedings of the National Academy of Sciences.

[21]  E. Bolthausen An Iterative Construction of Solutions of the TAP Equations for the Sherrington–Kirkpatrick Model , 2012, 1201.2891.

[22]  Yanting Ma,et al.  Approximate Message Passing Algorithm With Universal Denoising and Gaussian Mixture Learning , 2015, IEEE Transactions on Signal Processing.

[23]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[24]  Joel A. Tropp,et al.  Living on the edge: phase transitions in convex programs with random data , 2013, 1303.6672.

[25]  Richard G. Baraniuk,et al.  From Denoising to Compressed Sensing , 2014, IEEE Transactions on Information Theory.

[26]  Benjamin Recht,et al.  Sharp Time–Data Tradeoffs for Linear Inverse Problems , 2015, IEEE Transactions on Information Theory.

[27]  Sundeep Rangan,et al.  Compressive phase retrieval via generalized approximate message passing , 2012, Allerton Conference.

[28]  Thierry Blu,et al.  Monte-Carlo Sure: A Black-Box Optimization of Regularization Parameters for General Denoising Algorithms , 2008, IEEE Transactions on Image Processing.

[29]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[30]  Yanting Ma,et al.  Analysis of approximate message passing with a class of non-separable denoisers , 2017, 2017 IEEE International Symposium on Information Theory (ISIT).

[31]  Florent Krzakala,et al.  Approximate message passing with restricted Boltzmann machine priors , 2015, ArXiv.

[32]  Ole Winther,et al.  A theory of solving TAP equations for Ising models with general invariant random matrices , 2015, ArXiv.

[33]  A. Montanari,et al.  Asymptotic mutual information for the balanced binary stochastic block model , 2016 .

[34]  Matan Gavish,et al.  Near-optimal matrix recovery from random linear measurements , 2018, Proceedings of the National Academy of Sciences.

[35]  Z. D. Bai,et al.  Necessary and Sufficient Conditions for Almost Sure Convergence of the Largest Eigenvalue of a Wigner Matrix , 1988 .

[36]  Andrea Montanari,et al.  Graphical Models Concepts in Compressed Sensing , 2010, Compressed Sensing.

[37]  Pablo A. Parrilo,et al.  Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization , 2007, SIAM Rev..

[38]  Ole Winther,et al.  Dynamical functional theory for compressed sensing , 2017, 2017 IEEE International Symposium on Information Theory (ISIT).

[39]  Andrea Montanari,et al.  The LASSO Risk for Gaussian Matrices , 2010, IEEE Transactions on Information Theory.

[40]  Philip Schniter,et al.  Compressive Imaging Using Approximate Message Passing and a Markov-Tree Prior , 2010, IEEE Transactions on Signal Processing.

[41]  Magda Peligrad,et al.  Almost-Sure Results for a Class of Dependent Random Variables , 1999 .

[42]  Michael Unser,et al.  Approximate Message Passing With Consistent Parameter Estimation and Applications to Sparse Learning , 2012, IEEE Transactions on Information Theory.

[43]  Sundeep Rangan,et al.  Vector approximate message passing , 2017, ISIT.

[44]  Andrea Montanari,et al.  Universality in Polytope Phase Transitions and Message Passing Algorithms , 2012, ArXiv.

[45]  Emmanuel J. Candès,et al.  Unbiased Risk Estimates for Singular Value Thresholding and Spectral Estimators , 2012, IEEE Transactions on Signal Processing.

[46]  Andrea Montanari,et al.  High dimensional robust M-estimation: asymptotic variance via approximate message passing , 2013, Probability Theory and Related Fields.