Convex optimization via inertial algorithms with vanishing Tikhonov regularization: fast convergence to the minimum norm solution

In a Hilbertian framework, for the minimization of a general convex differentiable function f , we introduce new inertial dynamics and algorithms that generate trajectories and iterates that converge fastly towards the minimizer of f with minimum norm. Our study is based on the nonautonomous version of the Polyak heavy ball method, which, at time t, is associated with the strongly convex function obtained by adding to f a Tikhonov regularization term with vanishing coefficient ǫ(t). In this dynamic, the damping coefficient is proportional to the square root of the Tikhonov regularization parameter ǫ(t). By adjusting the speed of convergence of ǫ(t) towards zero, we will obtain both rapid convergence towards the infimal value of f , and the strong convergence of the trajectories towards the element of minimum norm of the set of minimizers of f . In particular, we obtain an improved version of the dynamic of Su-Boyd-Candès for the accelerated gradient method of Nesterov. This study naturally leads to corresponding first-order algorithms obtained by temporal discretization. In the case of a proper lower semicontinuous and convex function f , we study the proximal algorithms in detail, and show that they benefit from similar properties.

[1]  H. Attouch,et al.  Asymptotic Control and Stabilization of Nonlinear Oscillators with Non-isolated Equilibria , 2002 .

[2]  Juan Peypouquet,et al.  Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity , 2018, Math. Program..

[3]  A Liapunov function approach to the stabilization of second order coupled systems , 2016, 1604.06547.

[4]  Hedy Attouch,et al.  Asymptotic stabilization of inertial gradient dynamics with time-dependent viscosity , 2017 .

[5]  Zaki Chbani,et al.  First-order optimization algorithms via inertial systems with Hessian driven damping , 2019, Mathematical Programming.

[6]  Alexandre Cabot,et al.  Proximal Point Algorithm Controlled by a Slowly Vanishing Term: Applications to Hierarchical Minimization , 2005, SIAM J. Optim..

[7]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[8]  S. Gadat,et al.  On the long time behavior of second order differential equations with asymptotically small dissipation , 2007, 0710.1107.

[9]  CONVERGENCE AND ASYMPTOTIC STABILIZATION FOR SOME DAMPED HYPERBOLIC EQUATIONS WITH NON-ISOLATED EQUILIBRIA , 2001 .

[10]  Jean-François Aujol,et al.  The Differential Inclusion Modeling FISTA Algorithm and Optimality of Convergence Rate in the Case b $\leq3$ , 2018, SIAM J. Optim..

[11]  R. Boţ,et al.  Forward-Backward and Tseng’s Type Penalty Schemes for Monotone Inclusion Problems , 2013, 1306.0352.

[12]  R. Cominetti,et al.  A Convergence Result for Non-Autonomous Subgradient Evolution Equations and Its Application to the Steepest Descent Exponential Penality Trajectory in Linear Programming , 2001 .

[13]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[14]  A. N. Tikhonov,et al.  Solutions of ill-posed problems , 1977 .

[15]  Felipe Alvarez,et al.  Asymptotic selection of viscosity equilibria of semilinear evolution equations by the introduction of a slowly vanishing term , 2006 .

[16]  H. Attouch,et al.  Asymptotic behavior of coupled dynamical systems with multiscale aspects , 2009, 0904.0397.

[17]  A. Cabot.  Inertial Gradient-Like Dynamical System Controlled by a Stabilizing Term , 2004 .

[18]  Asymptotic behavior of gradient-like dynamical systems involving inertia and multiscale aspects , 2016, 1602.00232.

[19]  A. Chambolle,et al.  On the convergence of the iterates of "FISTA" , 2015 .

[20]  Juan Peypouquet,et al.  Prox-Penalization and Splitting Methods for Constrained Variational Problems , 2011, SIAM J. Optim..

[21]  H. Attouch,et al.  A Dynamical Approach to Convex Minimization Coupling Approximation with the Steepest Descent Method , 1996 .

[22]  M. J. D. Powell,et al.  Nonlinear Programming—Sequential Unconstrained Minimization Techniques , 1969 .

[23]  A Tikhonov,et al.  Solution of Incorrectly Formulated Problems and the Regularization Method , 1963 .

[24]  H. Attouch,et al.  Fast convex optimization via inertial dynamics with Hessian driven damping , 2016, Journal of Differential Equations.

[25]  Juan Peypouquet,et al.  Coupling Forward-Backward with Penalty Schemes and Parallel Splitting for Constrained Variational Inequalities , 2011, SIAM J. Optim..

[26]  Hédy Attouch,et al.  Viscosity Solutions of Minimization Problems , 1996, SIAM J. Optim..

[27]  H. Attouch,et al.  Combining fast inertial dynamics for convex optimization with Tikhonov regularization , 2016, 1602.01973.

[28]  Hedy Attouch,et al.  The Rate of Convergence of Nesterov's Accelerated Forward-Backward Method is Actually Faster Than 1/k2 , 2015, SIAM J. Optim..

[29]  Patrick L. Combettes,et al.  A Parallel Splitting Method for Coupled Monotone Inclusions , 2009, SIAM J. Control. Optim..

[30]  R. Cominetti.  Coupling the Proximal Point Algorithm with Approximation Methods , 1997 .

[31]  Ramzi May,et al.  On an asymptotically autonomous system with Tikhonov type regularizing term , 2010 .

[32]  Y. Nesterov.  A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .

[33]  Patrick L. Combettes,et al.  A strongly convergent primal–dual method for nonoverlapping domain decomposition , 2016, Numerische Mathematik.

[34]  S. A. Hirstoaga.  Approximation et résolution de problèmes d'équilibre, de point fixe et d'inclusion monotone , 2006 .

[35]  Stephen P. Boyd,et al.  A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights , 2014, J. Mach. Learn. Res..

[36]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[37]  S. Sorin,et al.  Strong asymptotic convergence of evolution equations governed by maximal monotone operators with Tikhonov regularization , 2008 .