A convergent iterative hard thresholding for nonnegative sparsity optimization

The iterative hard thresholding (IHT) algorithm is a popular greedy-type method in (linear and nonlinear) compressed sensing and sparse optimization problems. In this paper, we give an improved iterative hard thresholding algorithm for solving the nonnegative sparsity optimization (NSO) by employing the Armijo-type stepsize rule, which automatically adjusts the stepsize and support set and leads to a sufficient decrease of the objective function each iteration. Consequently, the improved IHT algorithm enjoys several convergence properties under standard assumptions. Those include the convergence to $\alpha$-stationary point (also known as $L$-stationary point in literature if the objective function has Lipschitz gradient) and the finite identification of the true support set. We also characterize when the full sequence converges to a local minimizer of NSO and establish its linear convergence rate. Extensive numerical experiments are included to demonstrate the good performance of the proposed algorithm.

[1]  Coralia Cartis,et al.  A New and Improved Quantitative Recovery Analysis for Iterative Hard Thresholding Algorithms in Compressed Sensing , 2013, IEEE Transactions on Information Theory.

[2]  Adrian S. Lewis,et al.  Convex Analysis And Nonlinear Optimization , 2000 .

[3]  Heinz H. Bauschke,et al.  Restricted Normal Cones and Sparsity Optimization with Affine Constraints , 2012, Found. Comput. Math..

[4]  Deanna Needell,et al.  CoSaMP: Iterative signal recovery from incomplete and inaccurate samples , 2008, ArXiv.

[5]  Paul H. Calamai,et al.  Projected gradient methods for linearly constrained problems , 1987, Math. Program..

[6]  Martin J. Wainwright,et al.  A unified framework for high-dimensional analysis of $M$-estimators with decomposable regularizers , 2009, NIPS.

[7]  Volkan Cevher,et al.  Model-Based Compressive Sensing , 2008, IEEE Transactions on Information Theory.

[8]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[9]  Yonina C. Eldar,et al.  Sparsity Based Sub-wavelength Imaging with Partially Incoherent Light via Quadratic Compressed Sensing References and Links , 2022 .

[10]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[11]  Mike E. Davies,et al.  Normalized Iterative Hard Thresholding: Guaranteed Stability and Performance , 2010, IEEE Journal of Selected Topics in Signal Processing.

[12]  Xiaoming Yuan,et al.  A splitting method for separable convex programming , 2015 .

[13]  Jorge J. Moré,et al.  Computing a Trust Region Step , 1983 .

[14]  T. Blumensath.  Compressed Sensing with Nonlinear Observations , 2010 .

[15]  Lie Wang.  The L1L1 penalized LAD estimator for high dimensional linear regression , 2013, J. Multivar. Anal..

[16]  S. Mallat,et al.  Adaptive greedy approximations , 1997 .

[17]  Christian Kanzow,et al.  A QP-free constrained Newton-type method for variational inequality problems , 1999, Math. Program..

[18]  Naihua Xiu,et al.  Gradient Support Projection Algorithm for Affine Feasibility Problem with Sparsity and Nonnegativity , 2014, 1406.7178.

[19]  Nadav Hallak,et al.  On the Minimization Over Sparse Symmetric Sets: Projections, Optimality Conditions, and Algorithms , 2016, Math. Oper. Res..

[20]  Michael Elad,et al.  Sparse and Redundant Representations - From Theory to Applications in Signal and Image Processing , 2010 .

[21]  Olgica Milenkovic,et al.  Subspace Pursuit for Compressive Sensing Signal Reconstruction , 2008, IEEE Transactions on Information Theory.

[22]  Shuiwang Ji,et al.  SLEP: Sparse Learning with Efficient Projections , 2011 .

[23]  Ali Jalali,et al.  On Learning Discrete Graphical Models using Greedy Methods , 2011, NIPS.

[24]  Lawrence Carin,et al.  Sparse multinomial logistic regression: fast algorithms and generalization bounds , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Naihua Xiu,et al.  On Solutions of Sparsity Constrained Optimization , 2015 .

[26]  Mike E. Davies,et al.  Iterative Hard Thresholding for Compressed Sensing , 2008, ArXiv.

[27]  Qingshan Liu,et al.  Newton Greedy Pursuit: A Quadratic Approximation Method for Sparsity-Constrained Optimization , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Yonina C. Eldar,et al.  Sparsity Constrained Nonlinear Optimization: Optimality Conditions and Algorithms , 2012, SIAM J. Optim..

[29]  Xiao-Tong Yuan,et al.  Gradient Hard Thresholding Pursuit for Sparsity-Constrained Optimization , 2013, ICML.

[30]  Benar Fux Svaiter,et al.  Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods , 2013, Math. Program..

[31]  Donghui Chen,et al.  Nonnegativity constraints in numerical analysis , 2009, The Birth of Numerical Analysis.

[32]  Bhiksha Raj,et al.  Greedy sparsity-constrained optimization , 2011, 2011 Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers (ASILOMAR).

[33]  Yonina C. Eldar,et al.  GESPAR: Efficient Phase Retrieval of Sparse Signals , 2013, IEEE Transactions on Signal Processing.

[34]  T. Blumensath,et al.  Iterative Thresholding for Sparse Approximations , 2008 .

[35]  Yong Zhang,et al.  Sparse Approximation via Penalty Decomposition Methods , 2012, SIAM J. Optim..

[36]  Jeffrey D. Blanchard,et al.  CGIHT: Conjugate Gradient Iterative Hard Thresholding for Compressed Sensing and Matrix Completion , 2015 .

[37]  Zhaosong Lu,et al.  Optimization over Sparse Symmetric Sets via a Nonmonotone Projected Gradient Method , 2015, 1509.08581.

[38]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[39]  Simon Foucart,et al.  Hard Thresholding Pursuit: An Algorithm for Compressive Sensing , 2011, SIAM J. Numer. Anal..

[40]  Stephen P. Boyd,et al.  An Interior-Point Method for Large-Scale l1-Regularized Logistic Regression , 2007, J. Mach. Learn. Res..