The mutual information in random linear estimation

We consider the estimation of a signal from the knowledge of its noisy linear random Gaussian projections, a problem relevant in compressed sensing, sparse superposition codes or code division multiple access just to cite few. There has been a number of works considering the mutual information for this problem using the heuristic replica method from statistical physics. Here we put these considerations on a firm rigorous basis. First, we show, using a Guerra-type interpolation, that the replica formula yields an upper bound to the exact mutual information. Secondly, for many relevant practical cases, we present a converse lower bound via a method that uses spatial coupling, state evolution analysis and the I-MMSE theorem. This yields, in particular, a single letter formula for the mutual information and the minimal-mean-square error for random Gaussian linear estimation of all discrete bounded signals.

[1]  Nicolas Macris,et al.  On the Capacity of a Code Division Multiple Access System , 2007 .

[2]  M. Mézard.  The space of interactions in neural networks: Gardner's computation with the cavity method , 1989 .

[3]  Sergio Verdú,et al.  Randomly spread CDMA: asymptotics via statistical physics , 2005, IEEE Transactions on Information Theory.

[4]  Shlomo Shamai,et al.  Support Recovery With Sparsely Sampled Free Random Matrices , 2011, IEEE Transactions on Information Theory.

[5]  Andrew R. Barron,et al.  Toward fast reliable communication at rates near capacity with Gaussian noise , 2010, 2010 IEEE International Symposium on Information Theory.

[6]  Antonia Maria Tulino,et al.  Random Matrix Theory and Wireless Communications , 2004, Found. Trends Commun. Inf. Theory.

[7]  Florent Krzakala,et al.  Non-adaptive pooling strategies for detection of rare faulty items , 2013, 2013 IEEE International Conference on Communications Workshops (ICC).

[8]  E. Bolthausen.  An Iterative Construction of Solutions of the TAP Equations for the Sherrington–Kirkpatrick Model , 2012, 1201.2891.

[9]  Florent Krzakala,et al.  Statistical physics-based reconstruction in compressed sensing , 2011, ArXiv.

[10]  Toshiyuki Tanaka,et al.  A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors , 2002, IEEE Trans. Inf. Theory.

[11]  Shlomo Shamai,et al.  Spectral Efficiency of CDMA with Random Spreading , 1999, IEEE Trans. Inf. Theory.

[12]  Y. Kabashima.  A CDMA multiuser detection algorithm on the basis of belief propagation , 2003 .

[13]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[14]  S. Verdú,et al.  Multiuser Detection and Statistical Mechanics , 2003 .

[15]  E. Gardner,et al.  Optimal storage properties of neural network models , 1988 .

[16]  Nicolas Macris,et al.  Threshold saturation of spatially coupled sparse superposition codes for all memoryless channels , 2016, 2016 IEEE Information Theory Workshop (ITW).

[17]  Florent Krzakala,et al.  Replica analysis and approximate message passing decoder for superposition codes , 2014, 2014 IEEE International Symposium on Information Theory.

[18]  R. Palmer,et al.  Solution of 'Solvable model of a spin glass' , 1977 .

[19]  E. Gardner.  The space of interactions in neural network models , 1988 .

[20]  Shlomo Shamai,et al.  Mutual information and minimum mean-square error in Gaussian channels , 2004, IEEE Transactions on Information Theory.

[21]  Shlomo Shamai,et al.  Mutual information and MMSE in gaussian channels , 2004, International Symposium onInformation Theory, 2004. ISIT 2004. Proceedings..

[22]  RushCynthia,et al.  Capacity-Achieving Sparse Superposition Codes via Approximate Message Passing Decoding , 2017 .

[23]  Florent Krzakala,et al.  Probabilistic reconstruction in compressed sensing: algorithms, phase diagrams, and threshold achieving matrices , 2012, ArXiv.

[24]  W. B. Johnson,et al.  Extensions of Lipschitz mappings into Hilbert space , 1984 .

[25]  F. Guerra.  Course 5 – An Introduction to Mean Field Spin Glas Theory: Methods and Results , 2006 .

[26]  Nicolas Macris,et al.  Coupled graphical models and their thresholds , 2010, 2010 IEEE Information Theory Workshop.

[27]  Nicolas Macris,et al.  Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula , 2016, NIPS.

[28]  Florent Krzakala,et al.  Mutual information in rank-one matrix estimation , 2016, 2016 IEEE Information Theory Workshop (ITW).

[29]  George Atia,et al.  Boolean Compressed Sensing and Noisy Group Testing , 2009, IEEE Transactions on Information Theory.

[30]  Surya Ganguli,et al.  Statistical mechanics of compressed sensing. , 2010, Physical review letters.

[31]  M. Mézard,et al.  Spin Glass Theory and Beyond , 1987 .

[32]  Yoshiyuki Kabashima,et al.  Erratum: A typical reconstruction limit of compressed sensing based on Lp-norm minimization , 2009, ArXiv.

[33]  Nicolas Macris,et al.  On the concentration of the capacity for a code division multiple access system , 2007, 2007 IEEE International Symposium on Information Theory.

[34]  Nicolas Macris,et al.  Tight Bounds on the Capacity of Binary Input Random CDMA Systems , 2008, IEEE Transactions on Information Theory.

[35]  Andrea Montanari,et al.  Analysis of Belief Propagation for Non-Linear Problems: The Example of CDMA (or: How to Prove Tanaka's Formula) , 2006, 2006 IEEE Information Theory Workshop - ITW '06 Punta del Este.

[36]  Dongning Guo,et al.  A single-letter characterization of optimal noisy compressed sensing , 2009, 2009 47th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[37]  Andrea Montanari,et al.  Universality in Polytope Phase Transitions and Message Passing Algorithms , 2012, ArXiv.

[38]  Kamil Sh. Zigangirov,et al.  Time-varying periodic convolutional codes with low-density parity-check matrix , 1999, IEEE Trans. Inf. Theory.

[39]  Toshiyuki Tanaka,et al.  Improvement of BP-based CDMA multiuser detection by spatial coupling , 2011, 2011 IEEE International Symposium on Information Theory Proceedings.

[40]  Galen Reeves,et al.  The replica-symmetric prediction for compressed sensing with Gaussian matrices is exact , 2016, 2016 IEEE International Symposium on Information Theory (ISIT).

[41]  References , 1971 .

[42]  Florent Krzakala,et al.  Approximate Message-Passing Decoder and Capacity Achieving Sparse Superposition Codes , 2015, IEEE Transactions on Information Theory.

[43]  Rüdiger L. Urbanke,et al.  Threshold Saturation via Spatial Coupling: Why Convolutional LDPC Ensembles Perform So Well over the BEC , 2010, IEEE Transactions on Information Theory.

[44]  Nicolas Macris,et al.  Threshold Saturation for Spatially Coupled LDPC and LDGM Codes on BMS Channels , 2013, IEEE Transactions on Information Theory.

[45]  B. Eynard,et al.  Random matrices. , 2015, 1510.04430.

[46]  Sergio Verdú,et al.  Optimal Phase Transitions in Compressed Sensing , 2011, IEEE Transactions on Information Theory.

[47]  Andrea Montanari,et al.  The dynamics of message passing on dense graphs, with applications to compressed sensing , 2010, 2010 IEEE International Symposium on Information Theory.

[48]  Henry D. Pfister,et al.  The effect of spatial coupling on compressive sensing , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[49]  Sundeep Rangan,et al.  Asymptotic Analysis of MAP Estimation via the Replica Method and Compressed Sensing , 2009, NIPS.

[50]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[51]  Henry D. Pfister,et al.  A simple proof of threshold saturation for coupled scalar recursions , 2012, 2012 7th International Symposium on Turbo Codes and Iterative Information Processing (ISTC).

[52]  Nicolas Macris,et al.  Proof of threshold saturation for spatially coupled sparse superposition codes , 2016, 2016 IEEE International Symposium on Information Theory (ISIT).

[53]  Ramji Venkataramanan,et al.  Capacity-Achieving Sparse Superposition Codes via Approximate Message Passing Decoding , 2015, IEEE Transactions on Information Theory.