Parameter Instability Regimes in Sparse Proximal Denoising Programs

Compressed sensing theory explains why LASSO programs recover structured high-dimensional signals with minimax order-optimal error. Yet, the optimal choice of the program’s governing parameter is often unknown in practice. It is still unclear how variation of the governing parameter impacts recovery error in compressed sensing, which is otherwise provably stable and robust. We establish a novel notion of instability in LASSO programs when the measurement matrix is identity. This is the proximal denoising setup. We prove asymptotic cusp-like behaviour of the risk as a function of the parameter choice, and illustrate the theory with numerical simulations. For example, a 0.1% underestimate of a LASSO parameter can increase the error significantly; and a 50% underestimate can cause the error to increase by a factor of 109. We hope that revealing parameter instability regimes of LASSO programs helps to inform a practitioner’s choice.

[1]  Roman Vershynin,et al.  High-Dimensional Probability , 2018 .

[2]  Emmanuel J. Candès,et al.  How well can we estimate a sparse vector? , 2011, ArXiv.

[3]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[4]  Abbas Mehrabian,et al.  A simple tool for bounding the deviation of random matrices on geometric sets , 2016, ArXiv.

[5]  Andrea Montanari,et al.  The LASSO Risk for Gaussian Matrices , 2010, IEEE Transactions on Information Theory.

[6]  Patrick L. Combettes,et al.  Alternating proximal algorithm for blind image recovery , 2010, 2010 IEEE International Conference on Image Processing.

[7]  S. Chatterjee.  A new perspective on least squares under convex constraint , 2014, 1402.0830.

[8]  Yaniv Plan,et al.  Robust 1-bit Compressed Sensing and Sparse Logistic Regression: A Convex Programming Approach , 2012, IEEE Transactions on Information Theory.

[9]  Andrea Montanari,et al.  The dynamics of message passing on dense graphs, with applications to compressed sensing , 2010, 2010 IEEE International Symposium on Information Theory.

[10]  Y. Meyer,et al.  Image decompositions using bounded variation and generalized homogeneous Besov spaces , 2007 .

[11]  Yaniv Plan,et al.  Dimension Reduction by Random Hyperplane Tessellations , 2014, Discret. Comput. Geom..

[12]  Michael Elad,et al.  Sparse and Redundant Representation Modeling—What Next? , 2012, IEEE Signal Processing Letters.

[13]  P. Wojtaszczyk,et al.  Stability and Instance Optimality for Gaussian Measurements in Compressed Sensing , 2010, Found. Comput. Math..

[14]  Pierre C Bellec,et al.  Localized Gaussian width of $M$-convex hulls with applications to Lasso and convex aggregation , 2017, Bernoulli.

[15]  Michael P. Friedlander,et al.  Gauge Optimization and Duality , 2013, SIAM J. Optim..

[16]  Joel A. Tropp,et al.  Living on the edge: phase transitions in convex programs with random data , 2013, 1303.6672.

[17]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[18]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[19]  Mee Young Park,et al.  L1‐regularization path algorithm for generalized linear models , 2007 .

[20]  Holger Rauhut,et al.  A Mathematical Introduction to Compressive Sensing , 2013, Applied and Numerical Harmonic Analysis.

[21]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[22]  Trevor Hastie,et al.  Regularization Paths for Generalized Linear Models via Coordinate Descent. , 2010, Journal of statistical software.

[23]  R. Handel.  Probability in High Dimension , 2014 .

[24]  Michael P. Friedlander,et al.  Probing the Pareto Frontier for Basis Pursuit Solutions , 2008, SIAM J. Sci. Comput..

[25]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[26]  Ingrid Daubechies,et al.  Variational image restoration by means of wavelets: simultaneous decomposition , 2005 .

[27]  Babak Hassibi,et al.  Asymptotically Exact Denoising in Relation to Compressed Sensing , 2013, ArXiv.

[28]  A. Banerjee.  Convex Analysis and Optimization , 2006 .

[29]  C. Borell.  The Brunn-Minkowski inequality in Gauss space , 1975 .

[30]  Michael Elad,et al.  On the Role of Sparse and Redundant Representations in Image Processing , 2010, Proceedings of the IEEE.

[31]  Mathukumalli Vidyasagar,et al.  An Introduction to Compressed Sensing , 2019 .

[32]  Yves Meyer,et al.  Oscillating Patterns in Image Processing and Nonlinear Evolution Equations: The Fifteenth Dean Jacqueline B. Lewis Memorial Lectures , 2001 .

[33]  I. Ibragimov,et al.  Norms of Gaussian sample functions , 1976 .

[34]  R. Adler,et al.  Random Fields and Geometry , 2007 .

[35]  D. Donoho,et al.  Sparse MRI: The application of compressed sensing for rapid MR imaging , 2007, Magnetic resonance in medicine.

[36]  Andrea Montanari,et al.  The distribution of the Lasso: Uniform control over sparse balls and adaptive parameter tuning , 2018, The Annals of Statistics.

[37]  Yaniv Plan,et al.  The Generalized Lasso With Non-Linear Observations , 2015, IEEE Transactions on Information Theory.

[38]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[39]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[40]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.