Enhancing Sparsity by Reweighted ℓ1 Minimization

It is now well understood that (1) it is possible to reconstruct sparse signals exactly from what appear to be highly incomplete sets of linear measurements and (2) that this can be done by constrained ℓ1 minimization. In this paper, we study a novel method for sparse signal recovery that in many situations outperforms ℓ1 minimization in the sense that substantially fewer measurements are needed for exact recovery. The algorithm consists of solving a sequence of weighted ℓ1-minimization problems where the weights used for the next iteration are computed from the value of the current solution. We present a series of experiments demonstrating the remarkable performance and broad applicability of this algorithm in the areas of sparse signal recovery, statistical estimation, error correction and image processing. Interestingly, superior gains are also achieved when our method is applied to recover signals with assumed near-sparsity in overcomplete representations—not by reweighting the ℓ1 norm of the coefficient sequence as is common, but by reweighting the ℓ1 norm of the transformed object. An immediate consequence is the possibility of highly efficient data acquisition protocols by improving on a technique known as Compressive Sensing.

[1]  E. Schlossmacher An Iterative Technique for Absolute Deviations Curve Fitting , 1973 .

[2]  J. Claerbout,et al.  Robust Modeling With Erratic Data , 1973 .

[3]  P. Holland,et al.  Robust regression using iteratively reweighted least-squares , 1977 .

[4]  H. L. Taylor,et al.  Deconvolution with the l 1 norm , 1979 .

[5]  Peter J. Huber,et al.  Robust Statistics , 2005, Wiley Series in Probability and Statistics.

[6]  R. Yarlagadda,et al.  Fast Algorithms for lp Deconvolution , 1985 .

[7]  J. Bee Bednar,et al.  Fast algorithms for lpdeconvolution , 1985, IEEE Trans. Acoust. Speech Signal Process..

[8]  F. Santosa,et al.  Linear inversion of ban limit reflection seismograms , 1986 .

[9]  D. Donoho,et al.  Uncertainty principles and signal recovery , 1989 .

[10]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[11]  B. Logan,et al.  Signal recovery and the large sieve , 1992 .

[12]  M. Dahleh,et al.  Control of Uncertain Systems: A Linear Programming Approach , 1995 .

[13]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[14]  Yoram Bresler,et al.  A new algorithm for computing sparse solutions to linear inverse problems , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[15]  Bhaskar D. Rao,et al.  Sparse signal reconstruction from limited data using FOCUSS: a re-weighted minimum norm algorithm , 1997, IEEE Trans. Signal Process..

[16]  L. Vandenberghe,et al.  Optimal wire and transistor sizing for circuits with non-tree topology , 1997, ICCAD 1997.

[17]  Stephen P. Boyd,et al.  Optimal wire and transistor sizing for circuits with non-tree topology , 1997, 1997 Proceedings of IEEE International Conference on Computer Aided Design (ICCAD).

[18]  Guillermo Sapiro,et al.  Robust anisotropic diffusion , 1998, IEEE Trans. Image Process..

[19]  Stephen P. Boyd,et al.  Low-authority controller design via convex optimization , 1998, Proceedings of the 37th IEEE Conference on Decision and Control (Cat. No.98CH36171).

[20]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[21]  Stephen P. Boyd,et al.  Optimizing dominant time constant in RC circuits , 1998, IEEE Trans. Comput. Aided Des. Integr. Circuits Syst..

[22]  Tony F. Chan,et al.  Color TV: total variation methods for restoration of vector-valued images , 1998, IEEE Trans. Image Process..

[23]  Yoram Bresler,et al.  Globally convergent edge-preserving regularized reconstruction: an application to limited-angle tomography , 1998, IEEE Trans. Image Process..

[24]  Stephen P. Boyd,et al.  Low-Authority Controller Design by Means of Convex Optimization , 1999 .

[25]  Xiaoming Huo,et al.  Uncertainty principles and ideal atomic decomposition , 2001, IEEE Trans. Inf. Theory.

[26]  Michael Elad,et al.  A generalized uncertainty principle and sparse representation in pairs of bases , 2002, IEEE Trans. Inf. Theory.

[27]  Stephen P. Boyd,et al.  Log-det heuristic for matrix rank minimization with applications to Hankel and Euclidean distance matrices , 2003, Proceedings of the 2003 American Control Conference, 2003..

[28]  Rémi Gribonval,et al.  Sparse representations in unions of bases , 2003, IEEE Trans. Inf. Theory.

[29]  B. Ripley,et al.  Robust Statistics , 2018, Encyclopedia of Mathematical Geosciences.

[30]  D. Donoho,et al.  Redundant Multiscale Transforms and Their Application for Morphological Component Separation , 2004 .

[31]  Wotao Yin,et al.  Second-order Cone Programming Methods for Total Variation-Based Image Restoration , 2005, SIAM J. Sci. Comput..

[32]  Robert D. Nowak,et al.  A bound optimization approach to wavelet-based image deconvolution , 2005, IEEE International Conference on Image Processing 2005.

[33]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[34]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[35]  Richard G. Baraniuk,et al.  An Information-Theoretic Approach to Distributed Compressed Sensing ∗ , 2005 .

[36]  D. Donoho,et al.  Counting faces of randomly-projected polytopes when the projection radically lowers dimension , 2006, math/0607364.

[37]  Yaakov Tsaig,et al.  Extensions of compressed sensing , 2006, Signal Process..

[38]  D. Takhar,et al.  A compressed sensing camera : New theory and an implementation using digital micromirrors , 2006 .

[39]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[40]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[41]  Stephen P. Boyd,et al.  Growing Well-connected Graphs , 2006, Proceedings of the 45th IEEE Conference on Decision and Control.

[42]  H. Zou The Adaptive Lasso and Its Oracle Properties , 2006 .

[43]  Joel A. Tropp,et al.  Just relax: convex programming methods for identifying sparse signals in noise , 2006, IEEE Transactions on Information Theory.

[44]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[45]  Michael Elad,et al.  On the stability of the basis pursuit in the presence of noise , 2006, Signal Process..

[46]  J. Haupt,et al.  Compressive wireless sensing , 2006, 2006 5th International Conference on Information Processing in Sensor Networks.

[47]  Michael Elad,et al.  Analysis versus synthesis in signal priors , 2006, 2006 14th European Signal Processing Conference.

[48]  Yaakov Tsaig,et al.  Breakdown of equivalence between the minimal l1-norm solution and the sparsest solution , 2006, Signal Process..

[49]  Stephen P. Boyd,et al.  The Fastest Mixing Markov Process on a Graph and a Connection to a Maximum Variance Unfolding Problem , 2006, SIAM Rev..

[50]  Robert D. Nowak,et al.  Majorization–Minimization Algorithms for Wavelet-Based Image Restoration , 2007, IEEE Transactions on Image Processing.

[51]  Terence Tao,et al.  The Dantzig selector: Statistical estimation when P is much larger than n , 2005, math/0506081.

[52]  Rick Chartrand,et al.  Exact Reconstruction of Sparse Signals via Nonconvex Minimization , 2007, IEEE Signal Processing Letters.

[53]  Stephen P. Boyd,et al.  Portfolio optimization with linear and fixed transaction costs , 2007, Ann. Oper. Res..

[54]  David P. Wipf,et al.  A New View of Automatic Relevance Determination , 2007, NIPS.

[55]  D. Donoho,et al.  Sparse MRI: The application of compressed sensing for rapid MR imaging , 2007, Magnetic resonance in medicine.

[56]  H. Zou,et al.  One-step Sparse Estimates in Nonconcave Penalized Likelihood Models. , 2008, Annals of statistics.

[57]  Emmanuel J. Candès,et al.  Highly Robust Error Correction byConvex Programming , 2006, IEEE Transactions on Information Theory.

[58]  Wotao Yin,et al.  Bregman Iterative Algorithms for (cid:2) 1 -Minimization with Applications to Compressed Sensing ∗ , 2008 .

[59]  Rayan Saab,et al.  Stable sparse approximations via nonconvex optimization , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[60]  Wotao Yin,et al.  Iteratively reweighted algorithms for compressive sensing , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[61]  Stephen P. Boyd,et al.  1 Trend Filtering , 2009, SIAM Rev..