A Multi-step Inertial Forward-Backward Splitting Method for Non-convex Optimization

In this paper, we propose a multi-step inertial Forward--Backward splitting algorithm for minimizing the sum of two non-necessarily convex functions, one of which is proper lower semi-continuous while the other is differentiable with a Lipschitz continuous gradient. We first prove global convergence of the scheme with the help of the Kurdyka-{\L}ojasiewicz property. Then, when the non-smooth part is also partly smooth relative to a smooth submanifold, we establish finite identification of the latter and provide sharp local linear convergence analysis. The proposed method is illustrated on a few problems arising from statistics and machine learning.

[1]  Adrian S. Lewis,et al.  Active Sets, Nonsmoothness, and Sensitivity , 2002, SIAM J. Optim..

[2]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[3]  Adrian S. Lewis,et al.  Twice Differentiable Spectral Functions , 2001, SIAM J. Matrix Anal. Appl..

[4]  Adrian S. Lewis,et al.  The [barred L]ojasiewicz Inequality for Nonsmooth Subanalytic Functions with Applications to Subgradient Dynamical Systems , 2006, SIAM J. Optim..

[5]  Felipe Alvarez,et al.  On the Minimizing Property of a Second Order Dissipative System in Hilbert Spaces , 2000, SIAM J. Control. Optim..

[6]  J. Bolte,et al.  Characterizations of Lojasiewicz inequalities: Subgradient flows, talweg, convexity , 2009 .

[7]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[8]  Juan Peypouquet,et al.  A Dynamical Approach to an Inertial Forward-Backward Algorithm for Convex Minimization , 2014, SIAM J. Optim..

[9]  H. Attouch,et al.  An Inertial Proximal Method for Maximal Monotone Operators via Discretization of a Nonlinear Oscillator with Damping , 2001 .

[10]  Juan Peypouquet,et al.  Splitting Methods with Variable Metric for Kurdyka–Łojasiewicz Functions and General Convergence Rates , 2015, J. Optim. Theory Appl..

[11]  Mohamed-Jalal Fadili,et al.  Local Linear Convergence of Forward-Backward under Partial Smoothness , 2014, NIPS.

[12]  A. Daniilidis,et al.  Geometrical interpretation of the predictor-corrector type algorithms in structured optimization problems , 2006 .

[13]  Michael Elad,et al.  Stable recovery of sparse overcomplete representations in the presence of noise , 2006, IEEE Transactions on Information Theory.

[14]  A. Moudafi,et al.  Convergence of a splitting inertial proximal method for monotone operators , 2003 .

[15]  Bohua Zhan,et al.  Smooth Manifolds , 2021, Arch. Formal Proofs.

[16]  Robert E. Mahony,et al.  An Extrinsic Look at the Riemannian Hessian , 2013, GSI.

[17]  Mohamed-Jalal Fadili,et al.  Activity Identification and Local Linear Convergence of Forward-Backward-type Methods , 2015, SIAM J. Optim..

[18]  丸山 徹.  Convex Analysisの二,三の進展について , 1977 .

[19]  Boris Polyak.  Some methods of speeding up the convergence of iteration methods , 1964 .

[20]  Adrian S. Lewis,et al.  Alternating Projections on Manifolds , 2008, Math. Oper. Res..

[21]  A. Chambolle,et al.  On the Convergence of the Iterates of the “Fast Iterative Shrinkage/Thresholding Algorithm” , 2015, J. Optim. Theory Appl..

[22]  A. Lewis,et al.  Identifying active constraints via partial smoothness and prox-regularity , 2003 .

[23]  Thomas Brox,et al.  iPiano: Inertial Proximal Algorithm for Nonconvex Optimization , 2014, SIAM J. Imaging Sci..

[24]  Dirk A. Lorenz,et al.  An Inertial Forward-Backward Algorithm for Monotone Inclusions , 2014, Journal of Mathematical Imaging and Vision.

[25]  Dmitriy Drusvyatskiy,et al.  Optimality, identifiability, and sensitivity , 2012, Math. Program..

[26]  Charles Steinhorn,et al.  Tame Topology and O-Minimal Structures , 2008 .

[27]  Hai Yen Le.  Generalized subdifferentials of the rank function , 2013, Optim. Lett..

[28]  I. Chavel.  Riemannian Geometry: Subject Index , 2006 .

[29]  Adrian S. Lewis,et al.  Partial Smoothness, Tilt Stability, and Generalized Hessians , 2013, SIAM J. Optim..

[30]  Jérôme Malick,et al.  Newton methods for nonsmooth convex minimization: connections among -Lagrangian, Riemannian Newton and SQP methods , 2005, Math. Program..

[31]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[32]  Benar Fux Svaiter,et al.  Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods , 2013, Math. Program..

[33]  Radu Ioan Bot,et al.  An inertial forward–backward algorithm for the minimization of the sum of two nonconvex functions , 2014, EURO J. Comput. Optim..