Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity

In a Hilbert space setting $${{\mathcal {H}}}$$H, we study the fast convergence properties as $$t \rightarrow + \infty $$t→+∞ of the trajectories of the second-order differential equation $$\begin{aligned} \ddot{x}(t) + \frac{\alpha }{t} \dot{x}(t) + \nabla \Phi (x(t)) = g(t), \end{aligned}$$x¨(t)+αtx˙(t)+∇Φ(x(t))=g(t),where $$\nabla \Phi $$∇Φ is the gradient of a convex continuously differentiable function $$\Phi : {{\mathcal {H}}} \rightarrow {{\mathbb {R}}}, \alpha $$Φ:H→R,α is a positive parameter, and $$g: [t_0, + \infty [ \rightarrow {{\mathcal {H}}}$$g:[t0,+∞[→H is a small perturbation term. In this inertial system, the viscous damping coefficient $$\frac{\alpha }{t}$$αt vanishes asymptotically, but not too rapidly. For $$\alpha \ge 3$$α≥3, and $$\int _{t_0}^{+\infty } t \Vert g(t)\Vert dt < + \infty $$∫t0+∞t‖g(t)‖dt<+∞, just assuming that $${{\mathrm{argmin\,}}}\Phi \ne \emptyset $$argminΦ≠∅, we show that any trajectory of the above system satisfies the fast convergence property $$\begin{aligned} \Phi (x(t))- \min _{{{\mathcal {H}}}}\Phi \le \frac{C}{t^2}. \end{aligned}$$Φ(x(t))-minHΦ≤Ct2.Moreover, for $$\alpha > 3$$α>3, any trajectory converges weakly to a minimizer of $$\Phi $$Φ. The strong convergence is established in various practical situations. These results complement the $${{\mathcal {O}}}(t^{-2})$$O(t-2) rate of convergence for the values obtained by Su, Boyd and Candès in the unperturbed case $$g=0$$g=0. Time discretization of this system, and some of its variants, provides new fast converging algorithms, expanding the field of rapid methods for structured convex minimization introduced by Nesterov, and further developed by Beck and Teboulle with FISTA. This study also complements recent advances due to Chambolle and Dossal.

[1]  Ronald E. Bruck Asymptotic convergence of nonlinear contraction semigroups in Hilbert space , 1975 .

[2]  Y. Nesterov Gradient methods for minimizing composite objective function , 2007 .

[3]  A. Moudafi,et al.  Convergence of a splitting inertial proximal method for monotone operators , 2003 .

[4]  A. Chambolle,et al.  On the convergence of the iterates of "FISTA" , 2015 .

[5]  K. Knopp Theory and Application of Infinite Series , 1990 .

[6]  Giorgio C. Buttazzo,et al.  Variational Analysis in Sobolev and BV Spaces - Applications to PDEs and Optimization, Second Edition , 2014, MPS-SIAM series on optimization.

[7]  Mark W. Schmidt,et al.  Convergence Rates of Inexact Proximal-Gradient Methods for Convex Optimization , 2011, NIPS.

[8]  Juan Peypouquet,et al.  Asymptotic almost-equivalence of Lipschitz evolution systems in Banach spaces☆ , 2010 .

[9]  Donghwan Kim,et al.  Optimized first-order methods for smooth convex minimization , 2014, Mathematical Programming.

[10]  S. Gadat,et al.  On the long time behavior of second order differential equations with asymptotically small dissipation , 2007, 0710.1107.

[11]  H. Attouch,et al.  An Inertial Proximal Method for Maximal Monotone Operators via Discretization of a Nonlinear Oscillator with Damping , 2001 .

[12]  J. Peypouquet Convex Optimization in Normed Spaces , 2015 .

[13]  J. Bolte,et al.  A second-order gradient-like dissipative dynamical system with Hessian-driven damping.: Application to optimization and mechanics , 2002 .

[14]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[15]  Z. Opial Weak convergence of the sequence of successive approximations for nonexpansive mappings , 1967 .

[16]  J. Baillon,et al.  Un exemple concernant le comportement asymptotique de la solution du problème dudt + ∂ϑ(μ) ∋ 0 , 1978 .

[17]  Stephen P. Boyd,et al.  A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights , 2014, J. Mach. Learn. Res..

[18]  Ivan P. Gavrilyuk,et al.  Variational analysis in Sobolev and BV spaces , 2007, Math. Comput..

[19]  Felipe Alvarez,et al.  On the Minimizing Property of a Second Order Dissipative System in Hilbert Spaces , 2000, SIAM J. Control. Optim..

[20]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[21]  Y. Nesterov A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .

[22]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[23]  Jean-François Aujol,et al.  Stability of Over-Relaxations for the Forward-Backward Algorithm, Application to FISTA , 2015, SIAM J. Optim..

[24]  Sébastien Gadat,et al.  Second-order differential equations with asymptotically small dissipation and piecewise flat potentials , 2009 .

[25]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[26]  Dirk A. Lorenz,et al.  An Inertial Forward-Backward Algorithm for Monotone Inclusions , 2014, Journal of Mathematical Imaging and Vision.

[27]  Benar Fux Svaiter,et al.  Newton-Like Dynamics and Forward-Backward Methods for Structured Monotone Inclusions in Hilbert Spaces , 2014, J. Optim. Theory Appl..

[28]  Juan Peypouquet,et al.  A Dynamical Approach to an Inertial Forward-Backward Algorithm for Convex Minimization , 2014, SIAM J. Optim..

[29]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[30]  Osman Güler,et al.  New Proximal Point Algorithms for Convex Minimization , 1992, SIAM J. Optim..

[31]  S. Sorin,et al.  Evolution equations for maximal monotone operators: asymptotic analysis in continuous and discrete time , 2009, 0905.1270.

[32]  Luca Baldassarre,et al.  Accelerated and Inexact Forward-Backward Algorithms , 2013, SIAM J. Optim..

[33]  Juan Peypouquet,et al.  A unified approach to the asymptotic almost-equivalence of evolution systems without Lipschitz conditions , 2011 .

[34]  Massimo Gobbino,et al.  The Remarkable Effectiveness of Time-Dependent Damping Terms for Second Order Evolution Equations , 2016, SIAM J. Control. Optim..

[35]  Christian Wolf,et al.  Fast Exact Hyper-graph Matching with Dynamic Programming for Spatio-temporal Data , 2014, Journal of Mathematical Imaging and Vision.

[36]  J. Peypouquet Convex Optimization in Normed Spaces: Theory, Methods and Examples , 2015 .