On Compressive Sensing in Coding Problems: A Rigorous Approach

We take an information theoretic perspective on a classical sparse-sampling noisy linear model and present an analytical expression for the mutual information, which plays a central role in a variety of communications/signal processing problems. Such an expression was addressed previously by bounds, by simulations, and by the (nonrigorous) replica method. The expression of the mutual information is based on techniques used, addressing the minimum mean square error analysis. Using these expressions, we study specifically a variety of sparse linear communication models, which include coding in various settings, accounting also for multiple access channels, broadcast channels, and different wiretap problems. For those, we provide single-letter expressions and derive achievable rates, capturing the communications/signal processing features of these contemporary models.

[1]  Axthonv G. Oettinger,et al.  IEEE Transactions on Information Theory , 1998 .

[2]  Sriram Vishwanath,et al.  Secrecy using compressive sensing , 2011, 2011 IEEE Information Theory Workshop.

[3]  Dongning Guo,et al.  A single-letter characterization of optimal noisy compressed sensing , 2009, 2009 47th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[4]  Thomas M. Cover,et al.  Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing) , 2006 .

[5]  J. W. Silverstein,et al.  Spectral Analysis of Large Dimensional Random Matrices , 2009 .

[6]  Claude E. Shannon,et al.  Channels with Side Information at the Transmitter , 1958, IBM J. Res. Dev..

[7]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[8]  N. D. Bruijn.  Asymptotic methods in analysis , 1958 .

[9]  Galen Reeves,et al.  The Sampling Rate-Distortion Tradeoff for Sparsity Pattern Recovery in Compressed Sensing , 2010, IEEE Transactions on Information Theory.

[10]  Shlomo Shamai,et al.  Broadcast approach for the sparse-input random-sampled MIMO Gaussian channel , 2014, 2014 IEEE International Symposium on Information Theory.

[11]  Shlomo Shamai,et al.  On sparse sensing and sparse sampling of coded signals at sub-Landau rates , 2014, Trans. Emerg. Telecommun. Technol..

[12]  Emre Telatar,et al.  Capacity of Multi-antenna Gaussian Channels , 1999, Eur. Trans. Telecommun..

[13]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[14]  Sundeep Rangan,et al.  Asymptotic Analysis of MAP Estimation via the Replica Method and Applications to Compressed Sensing , 2009, IEEE Transactions on Information Theory.

[15]  Martin J. Wainwright,et al.  Information-Theoretic Limits on Sparsity Recovery in the High-Dimensional and Noisy Setting , 2007, IEEE Transactions on Information Theory.

[16]  Neri Merhav,et al.  Asymptotic MMSE analysis under sparse representation modeling , 2017, Signal Process..

[17]  Neri Merhav,et al.  Asymptotic MMSE analysis under sparse representation modeling , 2013, 2014 IEEE International Symposium on Information Theory.

[18]  Antonia Maria Tulino,et al.  Random Matrix Theory and Wireless Communications , 2004, Found. Trends Commun. Inf. Theory.

[19]  Carles Padró,et al.  Information Theoretic Security , 2013, Lecture Notes in Computer Science.

[20]  Antonia Maria Tulino,et al.  A statistical physics approach to the wiretap channel , 2013, 2013 IEEE International Symposium on Information Theory.

[21]  Sergio Verdú,et al.  A general formula for channel capacity , 1994, IEEE Trans. Inf. Theory.

[22]  R. Couillet,et al.  Random Matrix Methods for Wireless Communications: Estimation , 2011 .

[23]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[24]  Venkatesh Saligrama,et al.  Information Theoretic Bounds for Compressed Sensing , 2008, IEEE Transactions on Information Theory.

[25]  S. Varadhan,et al.  Large deviations , 2019, Graduate Studies in Mathematics.

[26]  Shlomo Shamai,et al.  Statistical Physics of Signal Estimation in Gaussian Noise: Theory and Examples of Phase Transitions , 2008, IEEE Transactions on Information Theory.

[27]  J R FRAZER,et al.  Methods analysis. , 1953, Journal of the American Dietetic Association.

[28]  Jacobus J. M. Verbaarschot,et al.  Critique of the replica trick , 1985 .

[29]  J. Nicholas Laneman,et al.  Information-spectrum methods for information-theoretic security , 2009, 2009 Information Theory and Applications Workshop.

[30]  Y. Rachlin,et al.  The secrecy of compressed sensing measurements , 2008, 2008 46th Annual Allerton Conference on Communication, Control, and Computing.

[31]  R. Palmer,et al.  The replica method and solvable spin glass model , 1979 .

[32]  L. Goddard.  Information Theory , 1962, Nature.

[33]  Shlomo Shamai,et al.  A broadcast approach for a single-user slowly fading MIMO channel , 2003, IEEE Trans. Inf. Theory.

[34]  Galen Reeves,et al.  Approximate Sparsity Pattern Recovery: Information-Theoretic Lower Bounds , 2010, IEEE Transactions on Information Theory.

[35]  Kamiar Rahnama Rad.  Nearly Sharp Sufficient Conditions on Exact Sparsity Pattern Recovery , 2009, IEEE Transactions on Information Theory.

[36]  J. N. Laneman,et al.  On the secrecy capacity of arbitrary wiretap channels , 2008, 2008 46th Annual Allerton Conference on Communication, Control, and Computing.

[37]  Martin J. Wainwright,et al.  Information-Theoretic Limits on Sparse Signal Recovery: Dense versus Sparse Measurement Matrices , 2008, IEEE Transactions on Information Theory.

[38]  Galen Reeves,et al.  A compressed sensing wire-tap channel , 2011, 2011 IEEE Information Theory Workshop.

[39]  A. D. Wyner,et al.  A Bound on the Number of Distinguishable Functions which are Time-Limited and Approximately Band-Limited , 1973 .

[40]  Sundeep Rangan,et al.  Necessary and Sufficient Conditions for Sparsity Pattern Recovery , 2008, IEEE Transactions on Information Theory.

[41]  Jun Muramatsu.  General formula for secrecy capacity of wiretap channel with noncausal state , 2014, 2014 IEEE International Symposium on Information Theory.

[42]  Thomas M. Cover,et al.  Network Information Theory , 2001 .

[43]  J. W. Silverstein,et al.  On the empirical distribution of eigenvalues of a class of large dimensional random matrices , 1995 .

[44]  Andrea Montanari,et al.  Message-passing algorithms for compressed sensing , 2009, Proceedings of the National Academy of Sciences.

[45]  Victor Dotsenko.  One more discussion of the replica trick: the example of the exact solution , 2010 .

[46]  Y. Bar-Shalom,et al.  Censoring sensors: a low-communication-rate scheme for distributed detection , 1996, IEEE Transactions on Aerospace and Electronic Systems.

[47]  Shlomo Shamai,et al.  Capacity of Channels With Frequency-Selective and Time-Selective Fading , 2010, IEEE Transactions on Information Theory.

[48]  M. Zirnbauer.  Another Critique of the Replica Trick , 1999, cond-mat/9903338.

[49]  Douglas L. Jones,et al.  Energy-efficient detection in sensor networks , 2005, IEEE Journal on Selected Areas in Communications.

[50]  Moe Z. Win,et al.  Asymptotic Performance of a Censoring Sensor Network , 2007, IEEE Transactions on Information Theory.

[51]  Shlomo Shamai,et al.  Support Recovery With Sparsely Sampled Free Random Matrices , 2011, IEEE Transactions on Information Theory.

[52]  Abbas El Gamal,et al.  Network Information Theory , 2021, 2021 IEEE 3rd International Conference on Advanced Trends in Information Theory (ATIT).

[53]  Sergio Verdú,et al.  Optimal Phase Transitions in Compressed Sensing , 2011, IEEE Transactions on Information Theory.

[54]  A. D. Wyner,et al.  The wire-tap channel , 1975, The Bell System Technical Journal.

[55]  Vahid Tarokh,et al.  Shannon-Theoretic Limits on Noisy Compressive Sampling , 2007, IEEE Transactions on Information Theory.

[56]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[57]  O. F. Cook.  The Method of Types , 1898 .

[58]  Neri Merhav.  Optimum Estimation via Gradients of Partition Functions and Information Measures: A Statistical-Mechanical Perspective , 2011, IEEE Transactions on Information Theory.

[59]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[60]  Neri Merhav,et al.  Statistical Physics and Information Theory , 2010, Found. Trends Commun. Inf. Theory.

[61]  Neri Merhav,et al.  Channel Coding in the Presence of Side Information , 2008, Found. Trends Commun. Inf. Theory.

[62]  K. Fernow.  New York , 1896, American Potato Journal.

[63]  Amir Dembo,et al.  Large Deviations Techniques and Applications , 1998 .

[64]  Bin Dai,et al.  Wiretap Channel With Side Information , 2006, ArXiv.