Solving Basis Pursuit

The problem of finding a minimum ℓ1-norm solution to an underdetermined linear system is an important problem in compressed sensing, where it is also known as basis pursuit. We propose a heuristic optimality check as a general tool for ℓ1-minimization, which often allows for early termination by “guessing” a primal-dual optimal pair based on an approximate support. Moreover, we provide an extensive numerical comparison of various state-of-the-art ℓ1-solvers that have been proposed during the last decade, on a large test set with a variety of explicitly given matrices and several right-hand sides per matrix reflecting different levels of solution difficulty. The results, as well as improvements by the proposed heuristic optimality check, are analyzed in detail to provide an answer to the question which algorithm is the best.

[1]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[2]  Philip Wolfe,et al.  Validation of subgradient optimization , 1974, Math. Program..

[3]  J. CandesE.,et al.  Robust uncertainty principles , 2006 .

[4]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[5]  Emmanuel J. Candès,et al.  NESTA: A Fast and Accurate First-Order Method for Sparse Recovery , 2009, SIAM J. Imaging Sci..

[6]  José Mario Martínez,et al.  Nonmonotone Spectral Projected Gradient Methods on Convex Sets , 1999, SIAM J. Optim..

[7]  Jian-Feng Cai,et al.  Linearized Bregman iterations for compressed sensing , 2009, Math. Comput..

[8]  Allen Y. Yang,et al.  A Review of Fast L(1)-Minimization Algorithms for Robust Face Recognition , 2010 .

[9]  Joel A. Tropp,et al.  Just relax: convex programming methods for identifying sparse signals in noise , 2006, IEEE Transactions on Information Theory.

[10]  Julien Mairal,et al.  Complexity Analysis of the Lasso Regularization Path , 2012, ICML.

[11]  Wotao Yin,et al.  Analysis and Generalizations of the Linearized Bregman Method , 2010, SIAM J. Imaging Sci..

[12]  S. Sastry,et al.  A Review of Fast `1-Minimization Algorithms for Robust Face Recognition , 2010 .

[13]  Antonio Frangioni,et al.  Convergence Analysis of Deflected Conditional Approximate Subgradient Methods , 2009, SIAM J. Optim..

[14]  Xiaoming Huo,et al.  Uncertainty principles and ideal atomic decomposition , 2001, IEEE Trans. Inf. Theory.

[15]  M. Patriksson,et al.  Conditional subgradient optimization -- Theory and applications , 1996 .

[16]  Joel A. Tropp,et al.  Signal Recovery From Random Measurements Via Orthogonal Matching Pursuit , 2007, IEEE Transactions on Information Theory.

[17]  Dirk A. Lorenz,et al.  Beyond convergence rates: exact recovery with the Tikhonov regularization with sparsity constraints , 2010, 1001.3276.

[18]  Roland Wunderling,et al.  Paralleler und objektorientierter Simplex-Algorithmus , 1996 .

[19]  Dirk A. Lorenz,et al.  Constructing Test Instances for Basis Pursuit Denoising , 2011, IEEE Transactions on Signal Processing.

[20]  M. R. Osborne,et al.  A new approach to variable selection in least squares problems , 2000 .

[21]  Franziska Wulf,et al.  Minimization Methods For Non Differentiable Functions , 2016 .

[22]  O. Scherzer,et al.  Necessary and sufficient conditions for linear convergence of ℓ1‐regularization , 2011 .

[23]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[24]  Yin Zhang,et al.  A Fast Algorithm for Sparse Reconstruction Based on Shrinkage, Subspace Optimization, and Continuation , 2010, SIAM J. Sci. Comput..

[25]  P. Camerini,et al.  On improving relaxation methods by modified gradient techniques , 1975 .

[26]  Dirk A. Lorenz,et al.  An infeasible-point subgradient method using adaptive approximate projections , 2011, Comput. Optim. Appl..

[27]  Yaakov Tsaig,et al.  Fast Solution of $\ell _{1}$ -Norm Minimization Problems When the Solution May Be Sparse , 2008, IEEE Transactions on Information Theory.

[28]  Dmitry M. Malioutov,et al.  Homotopy continuation for sparse signal representation , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[29]  Michael P. Friedlander,et al.  Probing the Pareto Frontier for Basis Pursuit Solutions , 2008, SIAM J. Sci. Comput..

[30]  A. Löbel.  Optimale Vehicle Scheduling in Public Transit , 1997 .

[31]  Wotao Yin,et al.  Bregman Iterative Algorithms for (cid:2) 1 -Minimization with Applications to Compressed Sensing ∗ , 2008 .

[32]  D. Donoho,et al.  Fast Solution of -Norm Minimization Problems When the Solution May Be Sparse , 2008 .

[33]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[34]  Charles Soussen,et al.  A sufficient condition on monotonic increase of the number of nonzero entry in the optimizer of L1 norm penalized least-square problem , 2011, ArXiv.

[35]  Zhifeng Zhang,et al.  Adaptive time-frequency decompositions , 1994 .

[36]  Dirk A. Lorenz,et al.  An Infeasible-Point Subgradient Method using Approximate Projections , 2011 .

[37]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[38]  Allen Y. Yang,et al.  Fast L1-Minimization Algorithms For Robust Face Recognition , 2010 .

[39]  Joel A. Tropp,et al.  Greed is good: algorithmic results for sparse approximation , 2004, IEEE Transactions on Information Theory.

[40]  Mário A. T. Figueiredo,et al.  Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems , 2007, IEEE Journal of Selected Topics in Signal Processing.

[41]  M. Salman Asif.  Primal dual pursuit: a homotopy based algorithm for the Dantzig selector , 2008 .

[42]  Junfeng Yang,et al.  Alternating Direction Algorithms for 1-Problems in Compressive Sensing , 2009, SIAM J. Sci. Comput..

[43]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[44]  Jean-Jacques Fuchs,et al.  On sparse representations in arbitrary redundant bases , 2004, IEEE Transactions on Information Theory.

[45]  Wotao Yin,et al.  Sparse Signal Reconstruction via Iterative Support Detection , 2009, SIAM J. Imaging Sci..

[46]  V. Klee,et al.  HOW GOOD IS THE SIMPLEX ALGORITHM , 1970 .

[47]  Hanif D. Sherali,et al.  Convergence and Computational Analyses for Some Variable Target Value and Subgradient Deflection Methods , 2006, Comput. Optim. Appl..

[48]  Ying Xiong.  Nonlinear Optimization , 2014 .