General inertial proximal gradient method for a class of nonconvex nonsmooth optimization problems

In this paper, we consider a general inertial proximal gradient method with constant and variable stepsizes for a class of nonconvex nonsmooth optimization problems. The proposed method incorporates two different extrapolations with respect to the previous iterates into the backward proximal step and the forward gradient step in classic proximal gradient method. Under more general parameter constraints, we prove that the proposed method generates a convergent subsequence and each limit point is a stationary point of the problem. Furthermore, the generated sequence is globally convergent to a stationary point if the objective function satisfies the Kurdyka–Łojasiewicz property. Local linear convergence also can be established for the proposed method with constant stepsizes by using a common error bound condition. In addition, we conduct some numerical experiments on nonconvex quadratic programming and SCAD penalty problems to demonstrate the advantage of the proposed method.

[1]  David L Donoho,et al.  Compressed sensing , 2006, IEEE Transactions on Information Theory.

[2]  Stephen J. Wright,et al.  Optimization for Machine Learning , 2013 .

[3]  Radu Ioan Bot,et al.  Inertial Douglas-Rachford splitting for monotone inclusion problems , 2014, Appl. Math. Comput..

[4]  Benar Fux Svaiter,et al.  Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods , 2013, Math. Program..

[5]  Radu Ioan Bot,et al.  An inertial forward–backward algorithm for the minimization of the sum of two nonconvex functions , 2014, EURO J. Comput. Optim..

[6]  K. Kurdyka.  On gradients of functions definable in o-minimal structures , 1998 .

[7]  Yurii Nesterov,et al.  Gradient methods for minimizing composite functions , 2012, Mathematical Programming.

[8]  Pierre Moulin,et al.  Local and global convergence of a general inertial proximal splitting scheme for minimizing composite functions , 2016, Comput. Optim. Appl..

[9]  Xin Chen,et al.  Sparse solutions to random standard quadratic optimization problems , 2013, Math. Program..

[10]  Lei Yang,et al.  Proximal Gradient Method with Extrapolation and Line Search for a Class of Nonconvex and Nonsmooth Problems , 2017, 1711.06831.

[11]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[12]  V. Koltchinskii,et al.  Nuclear norm penalization and optimal rates for noisy low rank matrix completion , 2010, 1011.6256.

[13]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[14]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[15]  R. Tyrrell Rockafellar,et al.  Convergence Rates in Forward-Backward Splitting , 1997, SIAM J. Optim..

[16]  Paul Tseng,et al.  A coordinate gradient descent method for nonsmooth separable minimization , 2008, Math. Program..

[17]  Thomas Brox,et al.  iPiano: Inertial Proximal Algorithm for Nonconvex Optimization , 2014, SIAM J. Imaging Sci..

[18]  Dirk A. Lorenz,et al.  An Inertial Forward-Backward Algorithm for Monotone Inclusions , 2014, Journal of Mathematical Imaging and Vision.

[19]  Marc Teboulle,et al.  Proximal alternating linearized minimization for nonconvex and nonsmooth problems , 2013, Mathematical Programming.

[20]  Radu Ioan Bot,et al.  An Inertial Tseng’s Type Proximal Algorithm for Nonsmooth and Nonconvex Optimization Problems , 2014, J. Optim. Theory Appl..

[21]  Boris Polyak.  Some methods of speeding up the convergence of iteration methods , 1964 .

[22]  Radu Ioan Bot,et al.  Proximal-gradient algorithms for fractional programming , 2016, Optimization.

[23]  Defeng Sun,et al.  Linear Rate Convergence of the Alternating Direction Method of Multipliers for Convex Composite Programming , 2017, Math. Oper. Res..

[24]  Yonina C. Eldar,et al.  Convex Optimization in Signal Processing and Communications , 2009 .

[25]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[26]  Juan Peypouquet,et al.  A Dynamical Approach to an Inertial Forward-Backward Algorithm for Convex Minimization , 2014, SIAM J. Optim..

[27]  H. Attouch,et al.  An Inertial Proximal Method for Maximal Monotone Operators via Discretization of a Nonlinear Oscillator with Damping , 2001 .

[28]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[29]  Hédy Attouch,et al.  Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Lojasiewicz Inequality , 2008, Math. Oper. Res..

[30]  Toshihide Ibaraki,et al.  Resource allocation problems - algorithmic approaches , 1988, MIT Press series in the foundations of computing.

[31]  Andrew E. B. Lim,et al.  Machine Learning and Portfolio Optimization , 2018, Manag. Sci..

[32]  Bo Wen,et al.  Linear Convergence of Proximal Gradient Algorithm with Extrapolation for a Class of Nonconvex Nonsmooth Minimization Problems , 2015, SIAM J. Optim..

[33]  Z.-Q. Luo,et al.  Error bounds and convergence analysis of feasible descent methods: a general approach , 1993, Ann. Oper. Res..

[34]  Deren Han,et al.  A note on the Douglas–Rachford splitting method for optimization problems involving hypoconvex functions , 2018, J. Glob. Optim..

[35]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[36]  Xiaoming Yuan,et al.  Local Linear Convergence of the Alternating Direction Method of Multipliers for Quadratic Programs , 2013, SIAM J. Numer. Anal..

[37]  A. Chambolle,et al.  On the Convergence of the Iterates of the “Fast Iterative Shrinkage/Thresholding Algorithm” , 2015, J. Optim. Theory Appl..

[38]  P. Tseng,et al.  On the linear convergence of descent methods for convex essentially smooth minimization , 1992 .

[39]  Paul-Emile Maingé,et al.  Convergence of One-Step Projected Gradient Methods for Variational Inequalities , 2016, J. Optim. Theory Appl..

[40]  Felipe Alvarez,et al.  On the Minimizing Property of a Second Order Dissipative System in Hilbert Spaces , 2000, SIAM J. Control. Optim..

[41]  A. Moudafi,et al.  Convergence of a splitting inertial proximal method for monotone operators , 2003 .

[42]  Marc Teboulle,et al.  First Order Methods beyond Convexity and Lipschitz Gradient Continuity with Applications to Quadratic Inverse Problems , 2017, SIAM J. Optim..

[43]  Thomas Brox,et al.  iPiasco: Inertial Proximal Algorithm for Strongly Convex Optimization , 2015, Journal of Mathematical Imaging and Vision.

[44]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[45]  Shiqian Ma,et al.  Inertial Proximal ADMM for Linearly Constrained Separable Convex Optimization , 2015, SIAM J. Imaging Sci..

[46]  R. Horst,et al.  DC Programming: Overview , 1999 .