Non-Convex Compressed Sensing with Training Data

Efficient algorithms for the sparse solution of under-determined linear systems Ax = b are known for matrices A satisfying suitable assumptions like the restricted isometry property (RIP). Without such assumptions little is known and without any assumptions on A the problem is NP -hard. A common approach is to replace `1 by `p minimization for 0 < p < 1, which is no longer convex and typically requires some form of local initial values for provably convergent algorithms. In this paper, we consider an alternative, where instead of suitable initial values we are provided with extra training problems Ax = Bl, l = 1, . . . , p that are related to our compressed sensing problem. They allow us to find the solution of the original problem Ax = b with high probability in the range of a one layer linear neural network with comparatively few assumptions on the matrix A.

[1]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[2]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[3]  Shuheng Zhou.  Sparse Hanson–Wright inequalities for subgaussian quadratic forms , 2015, Bernoulli.

[4]  Samet Oymak,et al.  Toward Moderate Overparameterization: Global Convergence Guarantees for Training Shallow Neural Networks , 2019, IEEE Journal on Selected Areas in Information Theory.

[5]  Wotao Yin,et al.  Iteratively reweighted algorithms for compressive sensing , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[6]  Rick Chartrand,et al.  Compressed sensing recovery via nonconvex shrinkage penalties , 2015, ArXiv.

[7]  Stefano Ermon,et al.  Modeling Sparse Deviations for Compressed Sensing using Generative Models , 2018, ICML.

[8]  Yinyu Ye,et al.  A note on the complexity of Lp minimization , 2011, Math. Program..

[9]  Alexandros G. Dimakis,et al.  Compressed Sensing with Deep Image Prior and Learned Regularization , 2018, ArXiv.

[10]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[11]  R. Chartrand,et al.  Restricted isometry properties and nonconvex compressive sensing , 2007 .

[12]  Song Han,et al.  Learning both Weights and Connections for Efficient Neural Network , 2015, NIPS.

[13]  Balas K. Natarajan,et al.  Sparse Approximate Solutions to Linear Systems , 1995, SIAM J. Comput..

[14]  Yiran Chen,et al.  Learning Structured Sparsity in Deep Neural Networks , 2016, NIPS.

[15]  Vladislav Voroninski,et al.  Global Guarantees for Enforcing Deep Generative Priors by Empirical Risk , 2017, IEEE Transactions on Information Theory.

[16]  John Wright,et al.  Complete Dictionary Recovery Over the Sphere II: Recovery by Riemannian Trust-Region Method , 2015, IEEE Transactions on Information Theory.

[17]  Oriol Vinyals,et al.  Qualitatively characterizing neural network optimization problems , 2014, ICLR.

[18]  Rémi Gribonval,et al.  Dictionary Identification - Sparse Matrix-Factorisation via ℓ _ 1 -Minimisation , 2020 .

[19]  Yi Ma,et al.  Complete Dictionary Learning via 𝓁4-Norm Maximization over the Orthogonal Group , 2019, J. Mach. Learn. Res..

[20]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[21]  Dmitry P. Vetrov,et al.  Variational Dropout Sparsifies Deep Neural Networks , 2017, ICML.

[22]  Reinhard Heckel,et al.  Compressive sensing with un-trained neural networks: Gradient descent finds the smoothest approximation , 2020, ICML.

[23]  Sanjeev Arora,et al.  Simple, Efficient, and Neural Algorithms for Sparse Coding , 2015, COLT.

[24]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[25]  Wuzhen Shi,et al.  Deep networks for compressed image sensing , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[26]  Prateek Jain,et al.  Learning Sparsely Used Overcomplete Dictionaries , 2014, COLT.

[27]  Aditya Bhaskara,et al.  More Algorithms for Provable Dictionary Learning , 2014, ArXiv.

[28]  Qiyu Sun,et al.  Recovery of sparsest signals via ℓq-minimization , 2010, ArXiv.

[29]  Song Li,et al.  Restricted p–isometry property and its application for nonconvex compressive sensing , 2012, Adv. Comput. Math..

[30]  Ohad Shamir,et al.  Spurious Local Minima are Common in Two-Layer ReLU Neural Networks , 2017, ICML.

[31]  A. Bruckstein,et al.  On the uniqueness of overcomplete dictionaries, and a practical way to retrieve them , 2006 .

[32]  Song Han,et al.  Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding , 2015, ICLR.

[33]  John Wright,et al.  Complete Dictionary Recovery Over the Sphere I: Overview and the Geometric Picture , 2015, IEEE Transactions on Information Theory.

[34]  Daniel Soudry,et al.  No bad local minima: Data independent training error guarantees for multilayer neural networks , 2016, ArXiv.

[35]  David Steurer,et al.  Dictionary Learning and Tensor Decomposition via the Sum-of-Squares Method , 2014, STOC.

[36]  I. Daubechies,et al.  Iteratively reweighted least squares minimization for sparse recovery , 2008, 0807.0575.

[37]  Yuanzhi Li,et al.  A Convergence Theory for Deep Learning via Over-Parameterization , 2018, ICML.

[38]  Liwei Wang,et al.  Gradient Descent Finds Global Minima of Deep Neural Networks , 2018, ICML.

[39]  Ruosong Wang,et al.  Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks , 2019, ICML.

[40]  Shiva Prasad Kasiviswanathan,et al.  Restricted Isometry Property under High Correlations , 2019, ArXiv.

[41]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Yuanzhi Li,et al.  Backward Feature Correction: How Deep Learning Performs Deep Learning , 2020, ArXiv.

[43]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[44]  Morteza Mardani,et al.  Neural Proximal Gradient Descent for Compressive Imaging , 2018, NeurIPS.

[45]  Alexandros G. Dimakis,et al.  Compressed Sensing using Generative Models , 2017, ICML.

[46]  G. Welper,et al.  A Relaxation Argument for Optimization in Neural Networks and Non-Convex Compressed Sensing , 2020, ArXiv.

[47]  Karin Schnass,et al.  Local identification of overcomplete dictionaries , 2014, J. Mach. Learn. Res..

[48]  Timothy Lillicrap,et al.  Deep Compressed Sensing , 2019, ICML.

[49]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[50]  Reinhard Heckel,et al.  A Provably Convergent Scheme for Compressive Sensing Under Random Generative Priors , 2018, Journal of Fourier Analysis and Applications.

[51]  Rina Panigrahy,et al.  Sparse Matrix Factorization , 2013, ArXiv.

[52]  Chinmay Hegde,et al.  Algorithmic Guarantees for Inverse Imaging with Untrained Network Priors , 2019, NeurIPS.

[53]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[54]  Samy Bengio,et al.  Understanding deep learning requires rethinking generalization , 2016, ICLR.

[55]  James O' Neill.  An Overview of Neural Network Compression , 2020, ArXiv.

[56]  Holger Rauhut,et al.  A Mathematical Introduction to Compressive Sensing , 2013, Applied and Numerical Harmonic Analysis.

[57]  Sanjeev Arora,et al.  New Algorithms for Learning Incoherent and Overcomplete Dictionaries , 2013, COLT.

[58]  Wei Hu,et al.  A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks , 2018, ICLR.

[59]  Yuanzhi Li,et al.  Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data , 2018, NeurIPS.

[60]  Wotao Yin,et al.  Improved Iteratively Reweighted Least Squares for Unconstrained Smoothed 퓁q Minimization , 2013, SIAM J. Numer. Anal..

[61]  S. Foucart,et al.  Sparsest solutions of underdetermined linear systems via ℓq-minimization for 0 , 2009 .