Convergence rates for an inertial algorithm of gradient type associated to a smooth non-convex minimization

We investigate an inertial algorithm of gradient type in connection with the minimization of a nonconvex differentiable function. The algorithm is formulated in the spirit of Nesterov's accelerated convex gradient method. We show that the generated sequences converge to a critical point of the objective function, if a regularization of the objective function satisfies the Kurdyka-{\L}ojasiewicz property. Further, we provide convergence rates for the generated sequences and the function values formulated in terms of the {\L}ojasiewicz exponent.

[1]  A. Chambolle,et al.  On the Convergence of the Iterates of the “Fast Iterative Shrinkage/Thresholding Algorithm” , 2015, J. Optim. Theory Appl..

[2]  K. Kurdyka.  On gradients of functions definable in o-minimal structures , 1998 .

[3]  Thomas Brox,et al.  iPiano: Inertial Proximal Algorithm for Nonconvex Optimization , 2014, SIAM J. Imaging Sci..

[4]  Patrick L. Combettes,et al.  Quasi-Nonexpansive Iterations on the Affine Hull of Orbits: From Mann's Mean Value Algorithm to Inertial Methods , 2017, SIAM J. Optim..

[5]  Benar Fux Svaiter,et al.  Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods , 2013, Math. Program..

[6]  A. Haraux,et al.  Convergence of Solutions of Second-Order Gradient-Like Systems with Analytic Nonlinearities , 1998 .

[7]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[8]  H. Attouch,et al.  Fast convex optimization via inertial dynamics with Hessian driven damping , 2016, Journal of Differential Equations.

[9]  Dirk A. Lorenz,et al.  An Inertial Forward-Backward Algorithm for Monotone Inclusions , 2014, Journal of Mathematical Imaging and Vision.

[10]  Radu Ioan Bot,et al.  The Proximal Alternating Direction Method of Multipliers in the Nonconvex Setting: Convergence Analysis and Rates , 2018, Math. Oper. Res..

[11]  Ernö Robert Csetnek,et al.  Approaching Nonsmooth Nonconvex Optimization Problems Through First Order Dynamical Systems with Hidden Acceleration and Hessian Driven Damping Terms , 2016 .

[12]  Boris Polyak.  Some methods of speeding up the convergence of iteration methods , 1964 .

[13]  Adrian S. Lewis,et al.  Clarke Subgradients of Stratifiable Functions , 2006, SIAM J. Optim..

[14]  Hédy Attouch,et al.  On the convergence of the proximal algorithm for nonsmooth functions involving analytic features , 2008, Math. Program..

[15]  Benjamin Recht,et al.  Analysis and Design of Optimization Algorithms via Integral Quadratic Constraints , 2014, SIAM J. Optim..

[16]  H. Attouch,et al.  THE HEAVY BALL WITH FRICTION METHOD, I. THE CONTINUOUS DYNAMICAL SYSTEM: GLOBAL EXPLORATION OF THE LOCAL MINIMA OF A REAL-VALUED FUNCTION BY ASYMPTOTIC ANALYSIS OF A DISSIPATIVE DYNAMICAL SYSTEM , 2000 .

[17]  J. Bolte,et al.  Characterizations of Lojasiewicz inequalities: Subgradient flows, talweg, convexity , 2009 .

[18]  Ralph Chill,et al.  On the Łojasiewicz–Simon gradient inequality , 2003 .

[19]  Hédy Attouch,et al.  Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Lojasiewicz Inequality , 2008, Math. Oper. Res..

[20]  L. Simon.  Asymptotics for a class of non-linear evolution equations, with applications to geometric problems , 1983 .

[21]  H. Attouch,et al.  Rate of convergence of the Nesterov accelerated gradient method in the subcritical case α ≤ 3 , 2017, ESAIM: Control, Optimisation and Calculus of Variations.

[22]  Stephen P. Boyd,et al.  A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights , 2014, J. Mach. Learn. Res..

[23]  R. Boţ,et al.  Approaching nonsmooth nonconvex minimization through second-order proximal-gradient dynamical systems , 2017, 1711.06570.

[24]  L. Dries,et al.  Geometric categories and o-minimal structures , 1996 .

[25]  Émilie Chouzenoux,et al.  Variable Metric Forward–Backward Algorithm for Minimizing the Sum of a Differentiable Function and a Convex Function , 2013, Journal of Optimization Theory and Applications.

[26]  Y. Nesterov.  A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .

[27]  Euhanna Ghadimi,et al.  Global convergence of the Heavy-ball method for convex optimization , 2014, 2015 European Control Conference (ECC).

[28]  Juan Peypouquet,et al.  Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity , 2018, Math. Program..

[29]  J. Bolte,et al.  On damped second-order gradient systems , 2014, 1411.8005.

[30]  Peter Ochs,et al.  Local Convergence of the Heavy-Ball Method and iPiano for Non-convex Optimization , 2016, J. Optim. Theory Appl..

[31]  L. Rosasco,et al.  Convergence of the forward-backward algorithm: beyond the worst-case with the help of geometry , 2017, Mathematical Programming.

[32]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[33]  BolteJérôme,et al.  Proximal Alternating Minimization and Projection Methods for Nonconvex Problems , 2010 .

[34]  R. Boţ,et al.  A second-order dynamical approach with variable damping to nonconvex smooth minimization , 2018, Applicable analysis.

[35]  Hao Jiang,et al.  Non-Ergodic Convergence Analysis of Heavy-Ball Algorithms , 2018, AAAI.

[36]  R. Boţ,et al.  A forward-backward dynamical approach to the minimization of the sum of a nonsmooth convex with a smooth nonconvex function , 2015, 1507.01416.

[37]  Marc Teboulle,et al.  Proximal alternating linearized minimization for nonconvex and nonsmooth problems , 2013, Mathematical Programming.

[38]  Juan Peypouquet,et al.  Splitting Methods with Variable Metric for Kurdyka–Łojasiewicz Functions and General Convergence Rates , 2015, J. Optim. Theory Appl..

[39]  Aurélien Garivier,et al.  On the Complexity of Best-Arm Identification in Multi-Armed Bandit Models , 2014, J. Mach. Learn. Res..

[40]  S. K. Zavriev,et al.  Heavy-ball method in nonconvex optimization problems , 1993 .

[41]  Bruce W. Suter,et al.  From error bounds to the complexity of first-order descent methods for convex functions , 2015, Math. Program..

[42]  Adrian S. Lewis,et al.  The [barred L]ojasiewicz Inequality for Nonsmooth Subanalytic Functions with Applications to Subgradient Dynamical Systems , 2006, SIAM J. Optim..

[43]  H. Attouch,et al.  An Inertial Proximal Method for Maximal Monotone Operators via Discretization of a Nonlinear Oscillator with Damping , 2001 .

[44]  C. Dossal,et al.  Optimal rate of convergence of an ODE associated to the Fast Gradient Descent schemes for b>0 , 2017 .

[45]  Juan Peypouquet,et al.  A Dynamical Approach to an Inertial Forward-Backward Algorithm for Convex Minimization , 2014, SIAM J. Optim..

[46]  Jean-François Aujol,et al.  Convergence rate of inertial Forward–Backward algorithm beyond Nesterov’s rule , 2018, Mathematical Programming.

[47]  David Stutz.  IPIANO : INERTIAL PROXIMAL ALGORITHM FOR NON-CONVEX OPTIMIZATION , 2016 .

[48]  R. Boţ,et al.  Newton-Like Dynamics Associated to Nonconvex Optimization Problems , 2017, Nonsmooth Optimization and Its Applications.

[49]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[50]  Radu Ioan Bot,et al.  An inertial forward–backward algorithm for the minimization of the sum of two nonconvex functions , 2014, EURO J. Comput. Optim..

[51]  Radu Ioan Bot,et al.  Inertial Douglas-Rachford splitting for monotone inclusion problems , 2014, Appl. Math. Comput..

[52]  Jean-François Aujol,et al.  Optimal Convergence Rates for Nesterov Acceleration , 2018, SIAM J. Optim..

[53]  Guoyin Li,et al.  Calculus of the Exponent of Kurdyka–Łojasiewicz Inequality and Its Applications to Linear Convergence of First-Order Methods , 2016, Foundations of Computational Mathematics.