On the convergence of a class of inertial dynamical systems with Tikhonov regularization

We consider a class of inertial second order dynamical system with Tikhonov regularization, which can be applied to solving the minimization of a smooth convex function. Based on the appropriate choices of the parameters in the dynamical system, we first show that the function value along the trajectories converges to the optimal value, and prove that the convergence rate can be faster than $$o(1/t^2)$$ . Moreover, by constructing proper energy function, we prove that the trajectories strongly converges to a minimizer of the objective function of minimum norm. Finally, some numerical experiments have been conducted to illustrate the theoretical results.

[1]  R. Boţ,et al.  A second-order dynamical system with Hessian-driven damping and penalty term associated to variational inequalities , 2016, Optimization.

[2]  Francis R. Bach,et al.  Learning with Submodular Functions: A Convex Optimization Perspective , 2011, Found. Trends Mach. Learn..

[3]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[4]  Radu Ioan Bot,et al.  Tikhonov regularization of a second order dynamical system with Hessian driven damping , 2019, Math. Program..

[5]  Ernö Robert Csetnek,et al.  Approaching Nonsmooth Nonconvex Optimization Problems Through First Order Dynamical Systems with Hidden Acceleration and Hessian Driven Damping Terms , 2016 .

[6]  R. Boţ,et al.  A second-order dynamical approach with variable damping to nonconvex smooth minimization , 2018, Applicable analysis.

[7]  Julien Mairal,et al.  Structured sparsity through convex optimization , 2011, ArXiv.

[8]  H. Attouch,et al.  Asymptotic Control and Stabilization of Nonlinear Oscillators with Non-isolated Equilibria , 2002 .

[9]  Juan Peypouquet,et al.  Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity , 2018, Math. Program..

[10]  Aurélien Garivier,et al.  On the Complexity of Best-Arm Identification in Multi-Armed Bandit Models , 2014, J. Mach. Learn. Res..

[11]  H. Attouch,et al.  Fast convex optimization via inertial dynamics with Hessian driven damping , 2016, Journal of Differential Equations.

[12]  Stephen P. Boyd,et al.  A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights , 2014, J. Mach. Learn. Res..

[13]  E. R. Csetnek.  Continuous Dynamics Related to Monotone Inclusions and Non-Smooth Optimization Problems , 2020, 2007.00460.

[14]  Laurent Condat,et al.  A Primal–Dual Splitting Method for Convex Optimization Involving Lipschitzian, Proximable and Linear Composite Terms , 2012, Journal of Optimization Theory and Applications.

[15]  H. Attouch,et al.  Fast convex optimization via time scaling of damped inertial gradient dynamics , 2020 .

[16]  H. Attouch,et al.  A Dynamical Approach to Convex Minimization Coupling Approximation with the Steepest Descent Method , 1996 .

[17]  Geoffrey E. Hinton,et al.  On the importance of initialization and momentum in deep learning , 2013, ICML.

[18]  H. Attouch,et al.  Combining fast inertial dynamics for convex optimization with Tikhonov regularization , 2016, 1602.01973.

[19]  H. Attouch,et al.  Rate of convergence of the Nesterov accelerated gradient method in the subcritical case α ≤ 3 , 2017, ESAIM: Control, Optimisation and Calculus of Variations.

[20]  Eduardo D. Sontag,et al.  Mathematical Control Theory: Deterministic Finite Dimensional Systems , 1990 .

[21]  Emmanuel J. Candès,et al.  NESTA: A Fast and Accurate First-Order Method for Sparse Recovery , 2009, SIAM J. Imaging Sci..

[22]  Stephen J. Wright,et al.  Optimization for Machine Learning , 2013 .

[23]  R. Boţ,et al.  Second-order dynamical systems with penalty terms associated to monotone inclusions , 2017, Analysis and Applications.

[24]  J. Bolte.  Continuous Gradient Projection Method in Hilbert Spaces , 2003 .

[25]  Hans De Sterck,et al.  Nesterov acceleration of alternating least squares for canonical tensor decomposition: Momentum step size selection and restart mechanisms , 2018, Numer. Linear Algebra Appl..

[26]  Hedy Attouch,et al.  Fast Proximal Methods via Time Scaling of Damped Inertial Dynamics , 2019, SIAM J. Optim..

[27]  Radu Ioan Bot,et al.  Second Order Forward-Backward Dynamical Systems For Monotone Inclusion Problems , 2015, SIAM J. Control. Optim..

[28]  John Wright,et al.  Robust Principal Component Analysis: Exact Recovery of Corrupted Low-Rank Matrices via Convex Optimization , 2009, NIPS.

[29]  Michael I. Jordan,et al.  A Dynamical Systems Perspective on Nesterov Acceleration , 2019, ICML.