The replica-symmetric prediction for compressed sensing with Gaussian matrices is exact

This paper considers the fundamental limit of compressed sensing for i.i.d. signal distributions and i.i.d. Gaussian measurement matrices. Its main contribution is a rigorous characterization of the asymptotic mutual information (MI) and minimum mean-square error (MMSE) in this setting. Under mild technical conditions, our results show that the limiting MI and MMSE are equal to the values predicted by the replica method from statistical physics. This resolves a well-known problem that has remained open for over a decade.

[1]  R. Gray.  Entropy and Information Theory , 1990, Springer New York.

[2]  Florent Krzakala,et al.  Statistical physics-based reconstruction in compressed sensing , 2011, ArXiv.

[3]  Galen Reeves,et al.  The Sampling Rate-Distortion Tradeoff for Sparsity Pattern Recovery in Compressed Sensing , 2010, IEEE Transactions on Information Theory.

[4]  Kellen Petersen August.  Real Analysis , 2009 .

[5]  Galen Reeves,et al.  Compressed sensing phase transitions: Rigorous bounds versus replica predictions , 2012, 2012 46th Annual Conference on Information Sciences and Systems (CISS).

[6]  D. Pollard.  A User's Guide to Measure Theoretic Probability by David Pollard , 2001 .

[7]  Shlomo Shamai,et al.  Support recovery with sparsely sampled free random matrices , 2011, ISIT.

[8]  Sergio Verdú,et al.  MMSE Dimension , 2010, IEEE Transactions on Information Theory.

[9]  Shlomo Shamai,et al.  Estimation in Gaussian Noise: Properties of the Minimum Mean-Square Error , 2010, IEEE Transactions on Information Theory.

[10]  Constantin P. Niculescu,et al.  Convex Functions and Their Applications: A Contemporary Approach , 2005 .

[11]  W. Rudin.  Principles of mathematical analysis , 1964 .

[12]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[13]  Shlomo Shamai,et al.  Spectral Efficiency of CDMA with Random Spreading , 1999, IEEE Trans. Inf. Theory.

[14]  M. Mézard,et al.  Information, Physics, and Computation , 2009 .

[15]  Dongning Guo,et al.  A single-letter characterization of optimal noisy compressed sensing , 2009, 2009 47th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[16]  Gábor Lugosi,et al.  Concentration Inequalities - A Nonasymptotic Theory of Independence , 2013, Concentration Inequalities.

[17]  Galen Reeves,et al.  Conditional central limit theorems for Gaussian projections , 2016, 2017 IEEE International Symposium on Information Theory (ISIT).

[18]  Sergio Verdú,et al.  Randomly spread CDMA: asymptotics via statistical physics , 2005, IEEE Transactions on Information Theory.

[19]  Andrea Montanari,et al.  The Noise-Sensitivity Phase Transition in Compressed Sensing , 2010, IEEE Transactions on Information Theory.

[20]  K. Chung,et al.  Limit Distributions for Sums of Independent Random Variables. , 1955 .

[21]  Andrea Montanari,et al.  The dynamics of message passing on dense graphs, with applications to compressed sensing , 2010, ISIT.

[22]  Dongning Guo,et al.  Asymptotic Mean-Square Optimality of Belief Propagation for Sparse Linear Systems , 2006, 2006 IEEE Information Theory Workshop - ITW '06 Chengdu.

[23]  Richard G. Baraniuk,et al.  Bayesian Compressive Sensing Via Belief Propagation , 2008, IEEE Transactions on Signal Processing.

[24]  Shlomo Shamai,et al.  Mutual information and minimum mean-square error in Gaussian channels , 2004, IEEE Transactions on Information Theory.

[25]  S. Edwards,et al.  Theory of spin glasses , 1975 .

[26]  Sundeep Rangan,et al.  Asymptotic Analysis of MAP Estimation via the Replica Method and Applications to Compressed Sensing , 2009, IEEE Transactions on Information Theory.

[27]  David Tse,et al.  Linear Multiuser Receivers: Effective Interference, Effective Bandwidth and User Capacity , 1999, IEEE Trans. Inf. Theory.

[28]  Ralf R. Müller,et al.  Channel capacity and minimum probability of error in large dual antenna array systems with binary modulation , 2003, IEEE Trans. Signal Process..

[29]  Yoshiyuki Kabashima,et al.  Erratum: A typical reconstruction limit of compressed sensing based on Lp-norm minimization , 2009, ArXiv.

[30]  Daniel Pérez Palomar,et al.  Gradient of mutual information in linear vector Gaussian channels , 2005, ISIT.

[31]  Henry D. Pfister,et al.  The effect of spatial coupling on compressive sensing , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[32]  Sergio Verdú,et al.  Functional Properties of Minimum Mean-Square Error and Mutual Information , 2012, IEEE Transactions on Information Theory.

[33]  David B. Dunson,et al.  Quantifying uncertainty in variable selection with arbitrary matrices , 2015, 2015 IEEE 6th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP).

[34]  Toshiyuki Tanaka,et al.  A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors , 2002, IEEE Trans. Inf. Theory.

[35]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[36]  R. Esposito,et al.  On a Relation between Detection and Estimation in Decision Theory , 1968, Inf. Control..

[37]  Shlomo Shamai,et al.  Statistical Physics of Signal Estimation in Gaussian Noise: Theory and Examples of Phase Transitions , 2008, IEEE Transactions on Information Theory.

[38]  Neri Merhav,et al.  Asymptotic MMSE analysis under sparse representation modeling , 2017, Signal Process..

[39]  Adel Javanmard,et al.  Information-Theoretically Optimal Compressed Sensing via Spatial Coupling and Approximate Message Passing , 2011, IEEE Transactions on Information Theory.

[40]  Olivier Rioul,et al.  Information Theoretic Proofs of Entropy Power Inequalities , 2007, IEEE Transactions on Information Theory.

[41]  Helge Holden,et al.  The Kolmogorov–Riesz compactness theorem , 2009, 0906.4883.

[42]  A. Rukhin.  Matrix Variate Distributions , 1999, The Multivariate Normal Distribution.

[43]  Nicolas Macris,et al.  Tight Bounds on the Capacity of Binary Input Random CDMA Systems , 2008, IEEE Transactions on Information Theory.

[44]  A. J. Stam.  Some Inequalities Satisfied by the Quantities of Information of Fisher and Shannon , 1959, Inf. Control..

[45]  Andrea Montanari,et al.  Analysis of Belief Propagation for Non-Linear Problems: The Example of CDMA (or: How to Prove Tanaka's Formula) , 2006, 2006 IEEE Information Theory Workshop - ITW '06 Punta del Este.

[46]  Neri Merhav.  Optimum Estimation via Gradients of Partition Functions and Information Measures: A Statistical-Mechanical Perspective , 2011, IEEE Transactions on Information Theory.

[47]  Andrea Montanari,et al.  Universality in polytope phase transitions and iterative algorithms , 2012, 2012 IEEE International Symposium on Information Theory Proceedings.

[48]  Paul R. Milgrom,et al.  Envelope Theorems for Arbitrary Choice Sets , 2002 .

[49]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[50]  Alison L Gibbs,et al.  On Choosing and Bounding Probability Metrics , 2002, math/0209021.