Inexact primal–dual gradient projection methods for nonlinear optimization on convex set

ABSTRACT In this paper, we propose a novel primal–dual inexact gradient projection method for nonlinear optimization problems with convex-set constraint. This method only needs inexact computation of the projections onto the convex set for each iteration, consequently reducing the computational cost for projections per iteration. This feature is attractive especially for solving problems where the projections are computationally not easy to calculate. Global convergence guarantee and ergodic convergence rate of the optimality residual are provided under loose assumptions. We apply our proposed strategy to -ball constrained problems. Numerical results exhibit that our inexact gradient projection methods for solving -ball constrained problems are more efficient than the exact methods.

[1]  A. Fiacco.  A Finite Algorithm for Finding the Projection of a Point onto the Canonical Simplex of R " , 2009 .

[2]  Terence Tao,et al.  The Dantzig selector: Statistical estimation when P is much larger than n , 2005, math/0506081.

[3]  Laurent Condat,et al.  A Fast Projection onto the Simplex and the l 1 Ball , 2015 .

[4]  Joel A. Tropp,et al.  Living on the edge: phase transitions in convex programs with random data , 2013, 1303.6672.

[5]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[6]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[7]  J. B. Rosen.  The gradient projection method for nonlinear programming: Part II , 1961 .

[8]  Marc Teboulle,et al.  Fast Gradient-Based Algorithms for Constrained Total Variation Image Denoising and Deblurring Problems , 2009, IEEE Transactions on Image Processing.

[9]  J. B. Rosen.  The Gradient Projection Method for Nonlinear Programming. Part I. Linear Constraints , 1960 .

[10]  P. L. Combettes,et al.  Solving monotone inclusions via compositions of nonexpansive averaged operators , 2004 .

[11]  Michael Patriksson,et al.  A survey on the continuous nonlinear resource allocation problem , 2008, Eur. J. Oper. Res..

[12]  A. Ng.  Feature selection, L1 vs. L2 regularization, and rotational invariance , 2004, Twenty-first international conference on Machine learning - ICML '04.

[13]  J. Borwein,et al.  Two-Point Step Size Gradient Methods , 1988 .

[14]  Honglak Lee,et al.  Efficient L1 Regularized Logistic Regression , 2006, AAAI.

[15]  J. Moreau.  Proximité et dualité dans un espace hilbertien , 1965 .

[16]  José Mario Martínez,et al.  Algorithm 813: SPG—Software for Convex-Constrained Optimization , 2001, TOMS.

[17]  A. Goldstein.  Convex programming in Hilbert space , 1964 .

[18]  Marc Teboulle,et al.  Conditional Gradient Algorithmsfor Rank-One Matrix Approximations with a Sparsity Constraint , 2011, SIAM Rev..

[19]  Philip Wolfe,et al.  An algorithm for quadratic programming , 1956 .

[20]  Mário A. T. Figueiredo,et al.  Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems , 2007, IEEE Journal of Selected Topics in Signal Processing.

[21]  Benjamin Naumann,et al.  Computational Methods In Optimization A Unified Approach , 2016 .

[22]  J. M. Martínez,et al.  Inexact spectral projected gradient methods on convex sets , 2003 .

[23]  Michael P. Friedlander,et al.  Probing the Pareto Frontier for Basis Pursuit Solutions , 2008, SIAM J. Sci. Comput..

[24]  Sébastien Bubeck,et al.  Convex Optimization: Algorithms and Complexity , 2014, Found. Trends Mach. Learn..

[25]  CondatLaurent.  Fast projection onto the simplex and the $$\pmb {l}_\mathbf {1}$$l1 ball , 2016 .

[26]  E. Yaz.  Linear Matrix Inequalities In System And Control Theory , 1998, Proceedings of the IEEE.

[27]  Ya-Xiang Yuan,et al.  On the truncated conjugate gradient method , 2000, Math. Program..

[28]  Boris Polyak,et al.  Constrained minimization methods , 1966 .

[29]  Yurii Nesterov,et al.  Gradient methods for minimizing composite functions , 2012, Mathematical Programming.

[30]  L. Grippo,et al.  A nonmonotone line search technique for Newton's method , 1986 .

[31]  I. Jolliffe,et al.  A Modified Principal Component Technique Based on the LASSO , 2003 .

[32]  Stephen P. Boyd,et al.  Low-Authority Controller Design by Means of Convex Optimization , 1999 .

[33]  D. Bertsekas.  On the Goldstein-Levitin-Polyak gradient projection method , 1974, CDC 1974.

[34]  Ion Necoara,et al.  On the Convergence of Inexact Projection Primal First-Order Methods for Convex Minimization , 2018, IEEE Transactions on Automatic Control.

[35]  Yoram Singer,et al.  Efficient projections onto the l1-ball for learning in high dimensions , 2008, ICML '08.

[36]  Stephen P. Boyd,et al.  An Interior-Point Method for Large-Scale l1-Regularized Logistic Regression , 2007, J. Mach. Learn. Res..

[37]  R. Tibshirani,et al.  A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis. , 2009, Biostatistics.

[38]  Stephen P. Boyd,et al.  Linear Matrix Inequalities in Systems and Control Theory , 1994 .

[39]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[40]  José Mario Martínez,et al.  Nonmonotone Spectral Projected Gradient Methods on Convex Sets , 1999, SIAM J. Optim..

[41]  Mark W. Schmidt,et al.  Optimizing Costly Functions with Simple Constraints: A Limited-Memory Projected Quasi-Newton Algorithm , 2009, AISTATS.

[42]  Mihailo Stojnic,et al.  Strong thresholds for ℓ2/ℓ1-optimization in block-sparse compressed sensing , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[43]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[44]  Yurii Nesterov,et al.  Generalized Power Method for Sparse Principal Component Analysis , 2008, J. Mach. Learn. Res..

[45]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[46]  Joachim M. Buhmann,et al.  Expectation-maximization for sparse and non-negative PCA , 2008, ICML '08.

[47]  G. McCormick,et al.  The Gradient Projection Method under Mild Differentiability Conditions , 1972 .

[48]  E. M. L. Beale,et al.  Nonlinear Programming: A Unified Approach. , 1970 .

[49]  丸山 徹.  Convex Analysisの二,三の進展について , 1977 .

[50]  M. Stojnic.  Various thresholds for $\ell_1$-optimization in compressed sensing , 2009 .