Information-Theoretic Lower Bounds for Compressive Sensing With Generative Models

It has recently been shown that for compressive sensing, significantly fewer measurements may be required if the sparsity assumption is replaced by the assumption the unknown vector lies near the range of a suitably-chosen generative model. In particular, in (Bora <italic>et al.</italic>, 2017) it was shown roughly <inline-formula> <tex-math notation="LaTeX">$O(k\log L)$ </tex-math></inline-formula> random Gaussian measurements suffice for accurate recovery when the generative model is an <inline-formula> <tex-math notation="LaTeX">$L$ </tex-math></inline-formula>-Lipschitz function with bounded <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>-dimensional inputs, and <inline-formula> <tex-math notation="LaTeX">$O(kd \log w)$ </tex-math></inline-formula> measurements suffice when the generative model is a <inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>-input ReLU network with depth <inline-formula> <tex-math notation="LaTeX">$d$ </tex-math></inline-formula> and width <inline-formula> <tex-math notation="LaTeX">$w$ </tex-math></inline-formula>. In this paper, we establish corresponding algorithm-independent lower bounds on the sample complexity using tools from minimax statistical analysis. In accordance with the above upper bounds, our results are summarized as follows: (i) We construct an <inline-formula> <tex-math notation="LaTeX">$L$ </tex-math></inline-formula>-Lipschitz generative model capable of generating group-sparse signals, and show that the resulting necessary number of measurements is <inline-formula> <tex-math notation="LaTeX">$\Omega (k \log L)$ </tex-math></inline-formula>; (ii) Using similar ideas, we construct ReLU networks with high depth and/or high depth for which the necessary number of measurements scales as <inline-formula> <tex-math notation="LaTeX">$\Omega \left({kd \frac {\log w}{\log n}}\right)$ </tex-math></inline-formula> (with output dimension <inline-formula> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula>), and in some cases <inline-formula> <tex-math notation="LaTeX">$\Omega (kd \log w)$ </tex-math></inline-formula>. As a result, we establish that the scaling laws derived in (Bora <italic>et al.</italic>, 2017) are optimal or near-optimal in the absence of further assumptions.

[1]  Martin J. Wainwright,et al.  Sharp Thresholds for High-Dimensional and Noisy Sparsity Recovery Using $\ell _{1}$ -Constrained Quadratic Programming (Lasso) , 2009, IEEE Transactions on Information Theory.

[2]  Martin J. Wainwright,et al.  Information-Theoretic Limits on Sparsity Recovery in the High-Dimensional and Noisy Setting , 2007, IEEE Transactions on Information Theory.

[3]  Eric Price,et al.  Lower Bounds for Compressed Sensing with Generative Models , 2019, ArXiv.

[4]  Xiaohan Wei,et al.  On the statistical rate of nonlinear recovery in generative models with heavy-tailed data , 2019, ICML.

[5]  Galen Reeves,et al.  Approximate Sparsity Pattern Recovery: Information-Theoretic Lower Bounds , 2010, IEEE Transactions on Information Theory.

[6]  Xiao Wang,et al.  Depth-Width Trade-offs for ReLU Networks via Sharkovsky's Theorem , 2019, ICLR.

[7]  Emmanuel J. Candès,et al.  A Probabilistic and RIPless Theory of Compressed Sensing , 2010, IEEE Transactions on Information Theory.

[8]  Helmut Bölcskei,et al.  The universal approximation power of finite-width deep ReLU networks , 2018, ArXiv.

[9]  Matus Telgarsky,et al.  Benefits of Depth in Neural Networks , 2016, COLT.

[10]  Vahid Tarokh,et al.  Shannon-Theoretic Limits on Noisy Compressive Sampling , 2007, IEEE Transactions on Information Theory.

[11]  Alexandros G. Dimakis,et al.  Compressed Sensing with Deep Image Prior and Learned Regularization , 2018, ArXiv.

[12]  Chinmay Hegde,et al.  Solving Linear Inverse Problems Using Gan Priors: An Algorithm with Provable Guarantees , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[13]  Emmanuel J. Candès,et al.  On the Fundamental Limits of Adaptive Sensing , 2011, IEEE Transactions on Information Theory.

[14]  Cem Aksoylar,et al.  Sparse Signal Processing With Linear and Nonlinear Observations: A Unified Shannon-Theoretic Approach , 2017, IEEE Trans. Inf. Theory.

[15]  Martin J. Wainwright,et al.  High-Dimensional Statistics , 2019 .

[16]  Venkatesh Saligrama,et al.  Information Theoretic Bounds for Compressed Sensing , 2008, IEEE Transactions on Information Theory.

[17]  Matus Telgarsky,et al.  Representation Benefits of Deep Feedforward Networks , 2015, ArXiv.

[18]  Joel A. Tropp,et al.  Just relax: convex programming methods for identifying sparse signals in noise , 2006, IEEE Transactions on Information Theory.

[19]  Alexandros G. Dimakis,et al.  Compressed Sensing using Generative Models , 2017, ICML.

[20]  Holger Rauhut,et al.  A Mathematical Introduction to Compressive Sensing , 2013, Applied and Numerical Harmonic Analysis.

[21]  Adel Javanmard,et al.  Information-Theoretically Optimal Compressed Sensing via Spatial Coupling and Approximate Message Passing , 2011, IEEE Transactions on Information Theory.

[22]  Joel A. Tropp,et al.  Greed is good: algorithmic results for sparse approximation , 2004, IEEE Transactions on Information Theory.

[23]  Raman Arora,et al.  Understanding Deep Neural Networks with Rectified Linear Units , 2016, Electron. Colloquium Comput. Complex..

[24]  Joel A. Tropp,et al.  Living on the edge: phase transitions in convex programs with random data , 2013, 1303.6672.

[25]  Stefano Ermon,et al.  Modeling Sparse Deviations for Compressed Sensing using Generative Models , 2018, ICML.

[26]  Jian Wang,et al.  A Sharp Condition for Exact Support Recovery With Orthogonal Matching Pursuit , 2017, IEEE Transactions on Signal Processing.

[27]  Volkan Cevher,et al.  An Introductory Guide to Fano's Inequality with Applications in Statistical Estimation , 2019, Information-Theoretic Methods in Data Science.

[28]  Shirin Jalali,et al.  Solving Inverse Problems via Auto-Encoders , 2019, IEEE Journal on Selected Areas in Information Theory.

[29]  Xin Yuan,et al.  Using auto-encoders for solving ill-posed linear inverse problems , 2019, ArXiv.

[30]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[31]  Martin J. Wainwright,et al.  Distance-based and continuum Fano inequalities with applications to statistical estimation , 2013, ArXiv.

[32]  Volkan Cevher,et al.  Limits on Support Recovery With Probabilistic Models: An Information-Theoretic Framework , 2015, IEEE Transactions on Information Theory.

[33]  Volkan Cevher,et al.  Model-Based Compressive Sensing , 2008, IEEE Transactions on Information Theory.

[34]  Vladislav Voroninski,et al.  Global Guarantees for Enforcing Deep Generative Priors by Empirical Risk , 2017, IEEE Transactions on Information Theory.

[35]  Emmanuel J. Candès,et al.  How well can we estimate a sparse vector? , 2011, ArXiv.

[36]  Reinhard Heckel,et al.  Deep Decoder: Concise Image Representations from Untrained Non-convolutional Networks , 2018, ICLR.