A New Computational Method for the Sparsest Solutions to Systems of Linear Equations

The connection between the sparsest solution to an underdetermined system of linear equations and the weighted $\ell_1$-minimization problem is established in this paper. We show that seeking the sparsest solution to a linear system can be transformed to searching for the densest slack variable of the dual problem of weighted $\ell_1$-minimization with all possible choices of nonnegative weights. Motivated by this fact, a new reweighted $\ell_1$-algorithm for the sparsest solutions of linear systems, going beyond the framework of existing sparsity-seeking methods, is proposed in this paper. Unlike existing reweighted $\ell_1$-methods that are based on the weights defined directly in terms of iterates, the new algorithm computes a weight in dual space via certain convex optimization and uses such a weight to locate the sparsest solutions. It turns out that the new algorithm converges to the sparsest solutions of linear systems under some mild conditions that do not require the uniqueness of the sparsest so...

[1]  P. Holland,et al.  Robust regression using iteratively reweighted least-squares , 1977 .

[2]  M. Lai,et al.  An Unconstrained $\ell_q$ Minimization with $0q\leq1$ for Sparse Solution of Underdetermined Linear Systems , 2011 .

[3]  David L. Donoho,et al.  De-noising by soft-thresholding , 1995, IEEE Trans. Inf. Theory.

[4]  Weiyu Xu,et al.  Improved sparse recovery thresholds with two-step reweighted ℓ1 minimization , 2010, 2010 IEEE International Symposium on Information Theory.

[5]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[6]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[7]  Weiyu Xu,et al.  Weighted ℓ1 minimization for sparse recovery with prior information , 2009, 2009 IEEE International Symposium on Information Theory.

[8]  Justin K. Romberg,et al.  Fast and Accurate Algorithms for Re-Weighted $\ell _{1}$-Norm Minimization , 2012, IEEE Transactions on Signal Processing.

[9]  Ronald A. DeVore,et al.  Some remarks on greedy algorithms , 1996, Adv. Comput. Math..

[10]  C. Carathéodory.  Über den Variabilitätsbereich der Koeffizienten von Potenzreihen, die gegebene Werte nicht annehmen , 1907 .

[11]  Xiaoming Huo,et al.  Uncertainty principles and ideal atomic decomposition , 2001, IEEE Trans. Inf. Theory.

[12]  Duan Li,et al.  Reweighted 1-Minimization for Sparse Solutions to Underdetermined Linear Systems , 2012, SIAM J. Optim..

[13]  David L. Donoho,et al.  Sparse Solution Of Underdetermined Linear Equations By Stagewise Orthogonal Matching Pursuit , 2006 .

[14]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[15]  Mike E. Davies,et al.  Greedy algorithms for compressed sensing , 2012, Compressed Sensing.

[16]  Olvi L. Mangasarian,et al.  Machine Learning via Polyhedral Concave Minimization , 1996 .

[17]  Michael Elad,et al.  Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ1 minimization , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[18]  Stephen J. Wright,et al.  Computational Methods for Sparse Solution of Linear Inverse Problems , 2010, Proceedings of the IEEE.

[19]  Rick Chartrand,et al.  Exact Reconstruction of Sparse Signals via Nonconvex Minimization , 2007, IEEE Signal Processing Letters.

[20]  ChenXiaojun,et al.  Convergence of the reweighted ℓ1 minimization algorithm for ℓ2---ℓp minimization , 2014 .

[21]  Demba Ba,et al.  Convergence and Stability of a Class of Iteratively Re-weighted Least Squares Algorithms for Sparse Signal Recovery in the Presence of Noise. , 2013, IEEE transactions on signal processing : a publication of the IEEE Signal Processing Society.

[22]  Dmitry Malioutov,et al.  Iterative log thresholding , 2013, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[23]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[24]  Justin K. Romberg,et al.  Sparse Recovery of Streaming Signals Using $\ell_1$-Homotopy , 2013, IEEE Transactions on Signal Processing.

[25]  Deanna Needell,et al.  CoSaMP: Iterative signal recovery from incomplete and inaccurate samples , 2008, ArXiv.

[26]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[27]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[28]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[29]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[30]  Jean-Jacques Fuchs,et al.  On sparse representations in arbitrary redundant bases , 2004, IEEE Transactions on Information Theory.

[31]  Deanna Needell,et al.  Noisy signal recovery via iterative reweighted L1-minimization , 2009, 2009 Conference Record of the Forty-Third Asilomar Conference on Signals, Systems and Computers.

[32]  Joel A. Tropp,et al.  Just relax: convex programming methods for identifying sparse signals in noise , 2006, IEEE Transactions on Information Theory.

[33]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[34]  I. Daubechies,et al.  Iteratively reweighted least squares minimization for sparse recovery , 2008, 0807.0575.

[35]  Zhaosong Lu,et al.  Iterative reweighted minimization methods for $$l_p$$lp regularized unconstrained nonlinear programming , 2012, Math. Program..

[36]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[37]  Emery N. Brown,et al.  Convergence and Stability of Iteratively Re-weighted Least Squares Algorithms , 2014, IEEE Transactions on Signal Processing.

[38]  Deanna Needell,et al.  Signal Recovery From Incomplete and Inaccurate Measurements Via Regularized Orthogonal Matching Pursuit , 2007, IEEE Journal of Selected Topics in Signal Processing.

[39]  Joan L. Mitchell,et al.  JPEG: Still Image Data Compression Standard , 1992 .

[40]  O. Mangasarian,et al.  NONLINEAR PERTURBATION OF LINEAR PROGRAMS , 1979 .

[41]  I F Gorodnitsky,et al.  Neuromagnetic source imaging with FOCUSS: a recursive weighted minimum norm algorithm. , 1995, Electroencephalography and clinical neurophysiology.

[42]  David P. Wipf,et al.  Iterative Reweighted 1 and 2 Methods for Finding Sparse Solutions , 2010, IEEE J. Sel. Top. Signal Process..

[43]  S. Mallat.  A wavelet tour of signal processing , 1998 .

[44]  Joel A. Tropp,et al.  Greed is good: algorithmic results for sparse approximation , 2004, IEEE Transactions on Information Theory.

[45]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[46]  Robert D. Nowak,et al.  Majorization–Minimization Algorithms for Wavelet-Based Image Restoration , 2007, IEEE Transactions on Image Processing.

[47]  Justin K. Romberg,et al.  Fast and Accurate Algorithms for Re-Weighted L1-Norm Minimization , 2012, ArXiv.

[48]  Yun-Bin Zhao,et al.  RSP-Based Analysis for Sparsest and Least $\ell_1$-Norm Solutions to Underdetermined Linear Systems , 2013, IEEE Transactions on Signal Processing.

[49]  Olgica Milenkovic,et al.  Subspace Pursuit for Compressive Sensing Signal Reconstruction , 2008, IEEE Transactions on Information Theory.

[50]  Bhaskar D. Rao,et al.  Sparse Bayesian learning for basis selection , 2004, IEEE Transactions on Signal Processing.

[51]  R. DeVore,et al.  Compressed sensing and best k-term approximation , 2008 .

[52]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[53]  Michael Elad,et al.  Sparse and Redundant Representations - From Theory to Applications in Signal and Image Processing , 2010 .

[54]  Wotao Yin,et al.  Iteratively reweighted algorithms for compressive sensing , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[55]  Sabine Van Huffel,et al.  Two-level ℓ1 minimization for compressed sensing , 2015, Signal Process..

[56]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..