Universality in Polytope Phase Transitions and Message Passing Algorithms

We consider a class of nonlinear mappings FA,N in R N indexed by symmetric random matrices A ∈ R N×N with independent entries. Within spin glass theory, special cases of these mappings correspond to iterating the TAP equations and were studied by Erwin Bolthausen. Within information theory, they are known as ‘approximate message passing’ algorithms. We study the high-dimensional (large N) behavior of the iterates of F for polynomial functions F, and prove that it is universal, i.e. it depends only on the first two moments of the entries of A, under a subgaussian tail condition. As an application, we prove the universality of a certain phase transition arising in polytope geometry and compressed sensing. This solves –for a broad class of random projections– a conjecture by David Donoho and Jared Tanner.

[1]  S. Kirkpatrick,et al.  Solvable Model of a Spin-Glass , 1975 .

[2]  R. Palmer,et al.  Solution of 'Solvable model of a spin glass' , 1977 .

[3]  M. Mézard,et al.  Spin Glass Theory and Beyond , 1987 .

[4]  Rolf Schneider,et al.  Random projections of regular simplices , 1992, Discret. Comput. Geom..

[5]  C. Tracy,et al.  Introduction to Random Matrices , 1992, hep-th/9210073.

[6]  J. W. Silverstein,et al.  No eigenvalues outside the support of the limiting spectral distribution of large-dimensional sample covariance matrices , 1998 .

[7]  Jacqueline Scherpen,et al.  Proceedings of the 1999 Conference on Information Sciences and Systems , 1999 .

[8]  David L. Donoho,et al.  Neighborly Polytopes And Sparse Solution Of Underdetermined Linear Equations , 2005 .

[9]  D. Donoho,et al.  Neighborliness of randomly projected simplices in high dimensions. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[10]  D. Donoho,et al.  Sparse nonnegative solution of underdetermined linear equations by linear programming. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[11]  D. Donoho,et al.  Counting faces of randomly-projected polytopes when the projection radically lowers dimension , 2006, math/0607364.

[12]  David L. Donoho,et al.  High-Dimensional Centrally Symmetric Polytopes with Neighborliness Proportional to Dimension , 2006, Discret. Comput. Geom..

[13]  D. Lubinsky.  A Survey of Weighted Polynomial Approximation with Exponential Weights , 2007 .

[14]  David L. Donoho,et al.  Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing , 2009, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[15]  R. Adamczak,et al.  Restricted Isometry Property of Matrices with Independent Columns and Neighborly Polytopes by Random Sampling , 2009, 0904.4723.

[16]  J. W. Silverstein,et al.  Spectral Analysis of Large Dimensional Random Matrices , 2009 .

[17]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[18]  Yoshiyuki Kabashima,et al.  Erratum: A typical reconstruction limit of compressed sensing based on Lp-norm minimization , 2009, ArXiv.

[19]  Philip Schniter,et al.  Turbo reconstruction of structured sparse signals , 2010, 2010 44th Annual Conference on Information Sciences and Systems (CISS).

[20]  Felipe Cucker,et al.  Smoothed Analysis of Moore-Penrose Inversion , 2010, SIAM J. Matrix Anal. Appl..

[21]  Sundeep Rangan,et al.  Generalized approximate message passing for estimation with random linear mixing , 2010, 2011 IEEE International Symposium on Information Theory Proceedings.

[22]  Andrea Montanari,et al.  The Noise-Sensitivity Phase Transition in Compressed Sensing , 2010, IEEE Transactions on Information Theory.

[23]  Florent Krzakala,et al.  Statistical physics-based reconstruction in compressed sensing , 2011, ArXiv.

[24]  Andrea Montanari,et al.  The LASSO Risk for Gaussian Matrices , 2010, IEEE Transactions on Information Theory.

[25]  D. Donoho,et al.  Information-theoretically optimal compressed sensing via spatial coupling and approximate message passing , 2013, 2012 IEEE International Symposium on Information Theory Proceedings.

[26]  T. Tao,et al.  Random matrices: The Universality phenomenon for Wigner ensembles , 2012, 1202.0068.

[27]  Adel Javanmard,et al.  State Evolution for General Approximate Message Passing Algorithms, with Applications to Spatial Coupling , 2012, ArXiv.

[28]  E. Bolthausen.  An Iterative Construction of Solutions of the TAP Equations for the Sherrington–Kirkpatrick Model , 2012, 1201.2891.

[29]  Cedric E. Ginestet.  Spectral Analysis of Large Dimensional Random Matrices, 2nd edn , 2012 .

[30]  Sundeep Rangan,et al.  Compressive phase retrieval via generalized approximate message passing , 2012, Allerton Conference.

[31]  Sundeep Rangan,et al.  Asymptotic Analysis of MAP Estimation via the Replica Method and Applications to Compressed Sensing , 2009, IEEE Transactions on Information Theory.

[32]  Richard G. Baraniuk,et al.  Asymptotic Analysis of Complex LASSO via Complex Approximate Message Passing (CAMP) , 2011, IEEE Transactions on Information Theory.

[33]  Andrea Montanari,et al.  Accurate Prediction of Phase Transitions in Compressed Sensing via a Connection to Minimax Denoising , 2011, IEEE Transactions on Information Theory.