Improved convergence rates and trajectory convergence for primal-dual dynamical systems with vanishing damping

In this work, we approach the minimization of a continuously differentiable convex function under linear equality constraints by a second-order dynamical system with asymptotically vanishing damping term. The system is formulated in terms of the augmented Lagrangian associated to the minimization problem. We show fast convergence of the primaldual gap, the feasibility measure, and the objective function value along the generated trajectories. In case the objective function has Lipschitz continuous gradient, we show that the primal-dual trajectory asymptotically weakly converges to a primal-dual optimal solution of the underlying minimization problem. To the best of our knowledge, this is the first result which guarantees the convergence of the trajectory generated by a primal-dual dynamical system with asymptotic vanishing damping. Moreover, we will rediscover in case of the unconstrained minimization of a convex differentiable function with Lipschitz continuous gradient all convergence statements obtained in the literature for Nesterov’s accelerated gradient method.

[1]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[2]  Karl Henrik Johansson,et al.  Randomized optimal consensus of multi-agent systems , 2011, Autom..

[3]  Ronald E. Bruck Asymptotic convergence of nonlinear contraction semigroups in Hilbert space , 1975 .

[4]  Juan Peypouquet,et al.  Convergence of inertial dynamics and proximal algorithms governed by maximally monotone operators , 2017, Mathematical Programming.

[5]  Z. Opial Weak convergence of the sequence of successive approximations for nonexpansive mappings , 1967 .

[6]  Jean-François Aujol,et al.  Convergence rate of inertial Forward–Backward algorithm beyond Nesterov’s rule , 2018, Mathematical Programming.

[7]  Ramzi May Asymptotic for a second order evolution equation with convex potential and vanishing damping term , 2015, 1509.05598.

[8]  R. Boţ,et al.  Second-order dynamical systems with penalty terms associated to monotone inclusions , 2017, Analysis and Applications.

[9]  Xianlin Zeng,et al.  Dynamical Primal-Dual Accelerated Method with Applications to Network Optimization , 2019 .

[10]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[11]  Hedy Attouch Fast inertial proximal ADMM algorithms for convex structured optimization with linear constraint , 2020 .

[12]  Ritesh Madan,et al.  Distributed algorithms for maximum lifetime routing in wireless sensor networks , 2006, IEEE Global Telecommunications Conference, 2004. GLOBECOM '04..

[13]  Hédy Attouch,et al.  Convergence Rates of Inertial Forward-Backward Algorithms , 2018, SIAM J. Optim..

[14]  Radu Ioan Bot,et al.  Second Order Forward-Backward Dynamical Systems For Monotone Inclusion Problems , 2015, SIAM J. Control. Optim..

[15]  Benar Fux Svaiter,et al.  Newton-Like Dynamics and Forward-Backward Methods for Structured Monotone Inclusions in Hilbert Spaces , 2014, J. Optim. Theory Appl..

[16]  Petra Koenig,et al.  Dynamics Of Evolutionary Equations , 2016 .

[17]  H. Attouch,et al.  Rate of convergence of inertial gradient dynamics with time-dependent viscous damping coefficient , 2018 .

[18]  H. Attouch,et al.  THE HEAVY BALL WITH FRICTION METHOD, I. THE CONTINUOUS DYNAMICAL SYSTEM: GLOBAL EXPLORATION OF THE LOCAL MINIMA OF A REAL-VALUED FUNCTION BY ASYMPTOTIC ANALYSIS OF A DISSIPATIVE DYNAMICAL SYSTEM , 2000 .

[19]  H. Attouch,et al.  Fast convex optimization via inertial dynamics with Hessian driven damping , 2016, Journal of Differential Equations.

[20]  Mohamed Ali Jendoubi,et al.  Asymptotics for a second-order differential equation with nonautonomous damping and an integrable source term , 2015 .

[21]  Boris Polyak Some methods of speeding up the convergence of iteration methods , 1964 .

[22]  Zaki Chbani,et al.  First-order optimization algorithms via inertial systems with Hessian driven damping , 2019, Mathematical Programming.

[23]  R. Boţ,et al.  An inertial alternating direction method of multipliers , 2014, 1404.4582.

[24]  H. Attouch,et al.  Rate of convergence of the Nesterov accelerated gradient method in the subcritical case α ≤ 3 , 2017, ESAIM: Control, Optimisation and Calculus of Variations.

[25]  Y. Nesterov A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .

[26]  R. Rockafellar Monotone Operators and the Proximal Point Algorithm , 1976 .

[27]  Felipe Alvarez,et al.  On the Minimizing Property of a Second Order Dissipative System in Hilbert Spaces , 2000, SIAM J. Control. Optim..

[28]  H. Attouch,et al.  Fast convex optimization via time scaling of damped inertial gradient dynamics , 2020 .

[29]  Ya-Ping Fang,et al.  Convergence Rates of Inertial Primal-Dual Dynamical Methods for Separable Convex Optimization Problems , 2020, SIAM J. Control. Optim..

[30]  Jean-François Aujol,et al.  Stability of Over-Relaxations for the Forward-Backward Algorithm, Application to FISTA , 2015, SIAM J. Optim..

[31]  A. Chambolle,et al.  On the Convergence of the Iterates of the “Fast Iterative Shrinkage/Thresholding Algorithm” , 2015, J. Optim. Theory Appl..

[32]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[33]  H. Attouch,et al.  Fast convex optimization via a third-order in time evolution equation , 2020, Optimization.

[34]  Radu Ioan Bot,et al.  Inertial Douglas-Rachford splitting for monotone inclusion problems , 2014, Appl. Math. Comput..

[35]  Feng Liu,et al.  Distributed gradient algorithm for constrained optimization with application to load sharing in power systems , 2015, Syst. Control. Lett..

[36]  J. Bolte,et al.  A second-order gradient-like dissipative dynamical system with Hessian-driven damping.: Application to optimization and mechanics , 2002 .

[37]  H. Attouch,et al.  Fast convex optimization via inertial dynamics combining viscous and Hessian-driven damping with time rescaling , 2020, Evolution Equations & Control Theory.

[38]  Zaki Chbani,et al.  Fast Convergence of Dynamical ADMM via Time Scaling of Damped Inertial Dynamics , 2021, Journal of Optimization Theory and Applications.

[39]  R. Boţ,et al.  Conjugate Duality in Convex Optimization , 2010 .

[40]  Hedy Attouch,et al.  Fast Proximal Methods via Time Scaling of Damped Inertial Dynamics , 2019, SIAM J. Optim..

[41]  RockaJellm MONOTONE OPERATORS ASSOCIATED WITH SADDLE . FUNCTIONS AND MINIMAX PROBLEMS R . 1 ' , 2022 .

[42]  Radu Ioan Bot,et al.  Tikhonov regularization of a second order dynamical system with Hessian driven damping , 2019, Math. Program..

[43]  Stephen P. Boyd,et al.  A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights , 2014, J. Mach. Learn. Res..

[44]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[45]  Hedy Attouch,et al.  The Rate of Convergence of Nesterov's Accelerated Forward-Backward Method is Actually Faster Than 1/k2 , 2015, SIAM J. Optim..

[46]  Hedy Attouch,et al.  Augmented Lagrangian and Proximal Alternating Direction Methods of Multipliers in Hilbert spaces . Applications to Games , PDE ’ s and Control , 2019 .

[47]  Yurii Nesterov,et al.  Gradient methods for minimizing composite functions , 2012, Mathematical Programming.

[48]  Juan Peypouquet,et al.  Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity , 2018, Math. Program..

[49]  Hedy Attouch,et al.  Asymptotic stabilization of inertial gradient dynamics with time-dependent viscosity , 2017 .

[50]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[51]  Lihua Xie,et al.  Distributed Continuous-Time Algorithms for Nonsmooth Extended Monotropic Optimization Problems , 2018, SIAM J. Control. Optim..

[52]  Radu Ioan Bot,et al.  A primal-dual dynamical approach to structured convex minimization problems , 2019, 1905.08290.

[53]  Feng Liu,et al.  Initialization-free distributed algorithms for optimal resource allocation with feasibility constraints and application to economic dispatch of power systems , 2015, Autom..

[54]  Richard G. Baraniuk,et al.  Fast Alternating Direction Optimization Methods , 2014, SIAM J. Imaging Sci..

[55]  R. Tyrrell Rockafellar,et al.  Augmented Lagrangians and Applications of the Proximal Point Algorithm in Convex Programming , 1976, Math. Oper. Res..