An Overview on Algorithms for Sparse Recovery

Sparsity is a very powerful prior for the identification of signals from noisy indirect measurements. The recovery of the signal is usually performed by suitable linearly constrained optimizations with additional sparsity enforcing barriers. Depending on the specific formulation, one can produce a variety of different algorithms. In this Chapter the numerical realizations of such linear and nonlinear programs are reviewed and compared with respect to different noisy scenarios. With the intention of providing a useful guide to users, we provide a sound mathematical introduction with rigorous derivations and proofs of convergence of all algorithms and we illustrate in detail their behavior by extensive numerical experiments.

[1]  E. Candès,et al.  Astronomical image representation by the curvelet transform , 2003, Astronomy & Astrophysics.

[2]  Massimo Fornasier,et al.  Numerical Methods for Sparse Recovery , 2010 .

[3]  Joel A. Tropp,et al.  Greed is good: algorithmic results for sparse approximation , 2004, IEEE Transactions on Information Theory.

[4]  Yaakov Tsaig,et al.  Fast Solution of $\ell _{1}$ -Norm Minimization Problems When the Solution May Be Sparse , 2008, IEEE Transactions on Information Theory.

[5]  Richard Baraniuk,et al.  APPLICATION OF COMPRESSIVE SENSING TO THE DESIGN OF WIDEBAND SIGNAL ACQUISITION RECEIVERS , 2009 .

[6]  Robert D. Nowak,et al.  Compressive distilled sensing: Sparse recovery using adaptivity in compressive measurements , 2009, 2009 Conference Record of the Forty-Third Asilomar Conference on Signals, Systems and Computers.

[7]  Robert D. Nowak,et al.  An EM algorithm for wavelet-based image restoration , 2003, IEEE Trans. Image Process..

[8]  Massimo Fornasier,et al.  Damping Noise-Folding and Enhanced Support Recovery in Compressed Sensing , 2013, IEEE Transactions on Signal Processing.

[9]  Massimo Fornasier,et al.  Compressive Sensing and Structured Random Matrices , 2010 .

[10]  Fionn Murtagh,et al.  Fast communication , 2002 .

[11]  Bhaskar D. Rao,et al.  Sparse signal reconstruction from limited data using FOCUSS: a re-weighted minimum norm algorithm , 1997, IEEE Trans. Signal Process..

[12]  M. R. Osborne,et al.  A new approach to variable selection in least squares problems , 2000 .

[13]  Stephen P. Boyd,et al.  Graph Implementations for Nonsmooth Convex Programs , 2008, Recent Advances in Learning and Control.

[14]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[15]  Balas K. Natarajan,et al.  Sparse Approximate Solutions to Linear Systems , 1995, SIAM J. Comput..

[16]  P. Holland,et al.  Robust regression using iteratively reweighted least-squares , 1977 .

[17]  J. King,et al.  A minimal error conjugate gradient method for ill-posed problems , 1989 .

[18]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[19]  Curtis R. Vogel,et al.  Ieee Transactions on Image Processing Fast, Robust Total Variation{based Reconstruction of Noisy, Blurred Images , 2022 .

[20]  Clemens Zarzer.  On Tikhonov regularization with non-convex sparsity constraints , 2009 .

[21]  Thomas Brox,et al.  On Iteratively Reweighted Algorithms for Nonsmooth Nonconvex Optimization in Computer Vision , 2015, SIAM J. Imaging Sci..

[22]  T. Blumensath,et al.  Iterative Thresholding for Sparse Approximations , 2008 .

[23]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[24]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[25]  Kazufumi Ito,et al.  A variational approach to sparsity optimization based on Lagrange multiplier theory , 2013 .

[26]  M. Fornasier,et al.  Multilevel Preconditioning for Adaptive Sparse Optimization , 2009 .

[27]  Z. Opial.  Weak convergence of the sequence of successive approximations for nonexpansive mappings , 1967 .

[28]  Roman Vershynin,et al.  Introduction to the non-asymptotic analysis of random matrices , 2010, Compressed Sensing.

[29]  D. A. H. Jacobs,et al.  A Generalization of the Conjugate-Gradient Method to Solve Complex Systems , 1986 .

[30]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[31]  Stéphane Mallat,et al.  A Wavelet Tour of Signal Processing - The Sparse Way, 3rd Edition , 2008 .

[32]  Mike E. Davies,et al.  Iterative Hard Thresholding for Compressed Sensing , 2008, ArXiv.

[33]  Yonina C. Eldar,et al.  Noise Folding in Compressed Sensing , 2011, IEEE Signal Processing Letters.

[34]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[35]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[36]  L. Dixon,et al.  Finite Algorithms in Optimization and Data Analysis. , 1988 .

[37]  Massimo Fornasier,et al.  Iterative Thresholding Meets Free-Discontinuity Problems , 2009, Found. Comput. Math..

[38]  R. DeVore,et al.  A Simple Proof of the Restricted Isometry Property for Random Matrices , 2008 .

[39]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[40]  Rick Chartrand,et al.  Exact Reconstruction of Sparse Signals via Nonconvex Minimization , 2007, IEEE Signal Processing Letters.

[41]  Wotao Yin,et al.  Iteratively reweighted algorithms for compressive sensing , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[42]  慧 廣瀬.  A Mathematical Introduction to Compressive Sensing , 2015 .

[43]  Wotao Yin,et al.  Improved Iteratively Reweighted Least Squares for Unconstrained Smoothed 퓁q Minimization , 2013, SIAM J. Numer. Anal..

[44]  E. Candès,et al.  Near-ideal model selection by ℓ1 minimization , 2008, 0801.0345.

[45]  R. Chartrand,et al.  Restricted isometry properties and nonconvex compressive sensing , 2007 .

[46]  L. Landweber.  An iteration formula for Fredholm integral equations of the first kind with application to the axially symmetric potential flow about elongated bodies of revolution , 1951 .

[47]  V. Naumova,et al.  Minimization of multi-penalty functionals by alternating iterative thresholding and optimal parameter choices , 2014, 1403.6718.

[48]  I. Loris.  On the performance of algorithms for the minimization of ℓ1-penalized functionals , 2007, 0710.4082.

[49]  Robert D. Nowak,et al.  Distilled Sensing: Adaptive Sampling for Sparse Detection and Estimation , 2010, IEEE Transactions on Information Theory.

[50]  Dirk A. Lorenz,et al.  Minimization of Non-smooth, Non-convex Functionals by Iterative Thresholding , 2015, J. Optim. Theory Appl..

[51]  Deanna Needell,et al.  Noisy signal recovery via iterative reweighted L1-minimization , 2009, 2009 Conference Record of the Forty-Third Asilomar Conference on Signals, Systems and Computers.

[52]  Deanna Needell,et al.  CoSaMP: Iterative signal recovery from incomplete and inaccurate samples , 2008, ArXiv.

[53]  R. Ramlau,et al.  ON THE MINIMIZATION OF A TIKHONOV FUNCTIONAL WITH A NON-CONVEX SPARSITY CONSTRAINT , 2009 .

[54]  Mário A. T. Figueiredo,et al.  Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems , 2007, IEEE Journal of Selected Topics in Signal Processing.

[55]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[56]  S. Voronin.  Regularization of Linear Systems with Sparsity Constraints with Applications to Large Scale Inverse Problems , 2012 .