Quantifying the asymptotic linear convergence speed of Anderson Acceleration applied to ADMM

We explain how Anderson Acceleration (AA) speeds up the Alternating Direction Method of Multipliers (ADMM), for the case where ADMM by itself converges linearly. We do so by considering the spectral properties of the Jacobians of ADMM and a stationary version of AA evaluated at the fixed point, where the coefficients of the stationary version are computed such that its asymptotic linear convergence factor is optimal. Numerical tests show that this allows us to quantify the improved linear asymptotic convergence speed of AA-ADMM as compared to the convergence factor of ADMM used by itself. This way of estimating AA-ADMM convergence speed is useful because there are no known convergence bounds for AA with finite window size that would allow quantification of this improvement in asymptotic convergence speed.

[1]  Daniel P. Robinson,et al.  A Dynamical Systems Perspective on Nonsmooth Constrained Optimization , 2018 .

[2]  Daniel P. Robinson,et al.  ADMM and Accelerated ADMM as Continuous Dynamical Systems , 2018, ICML.

[3]  Mikael Johansson,et al.  Anderson Acceleration of Proximal Gradient Methods , 2020, ICML.

[4]  Konstantina Christakopoulou,et al.  Accelerated Alternating Direction Method of Multipliers , 2015, KDD.

[5]  Jacob K. White,et al.  GMRES-Accelerated ADMM for Quadratic Objectives , 2016, SIAM J. Optim..

[6]  Bailin Deng,et al.  Anderson acceleration for geometry optimization and physics simulation , 2018, ACM Trans. Graph..

[7]  Donald G. M. Anderson.  Iterative Procedures for Nonlinear Integral Equations , 1965, JACM.

[8]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[9]  Bailin Deng,et al.  Accelerating ADMM for efficient simulation and optimization , 2019, ACM Trans. Graph..

[10]  Richard G. Baraniuk,et al.  Fast Alternating Direction Optimization Methods , 2014, SIAM J. Imaging Sci..

[11]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..

[12]  Daniel Boley.  Linear Convergence of ADMM on a Model Problem , 2012 .

[13]  Hans De Sterck,et al.  Nesterov acceleration of alternating least squares for canonical tensor decomposition: Momentum step size selection and restart mechanisms , 2018, Numer. Linear Algebra Appl..

[14]  Zhi-Quan Luo,et al.  On the linear convergence of the alternating direction method of multipliers , 2012, Mathematical Programming.

[15]  Bingsheng He,et al.  On non-ergodic convergence rate of Douglas–Rachford alternating direction method of multipliers , 2014, Numerische Mathematik.

[16]  C. T. Kelley,et al.  Convergence Analysis for Anderson Acceleration , 2015, SIAM J. Numer. Anal..

[17]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[18]  Wotao Yin,et al.  Faster Convergence Rates of Relaxed Peaceman-Rachford and ADMM Under Regularity Assumptions , 2014, Math. Oper. Res..

[19]  Jingwei Liang,et al.  Trajectory of Alternating Direction Method of Multipliers and Adaptive Acceleration , 2019, NeurIPS.

[20]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[21]  Michael I. Jordan,et al.  A General Analysis of the Convergence of ADMM , 2015, ICML.

[22]  Homer F. Walker,et al.  Anderson Acceleration for Fixed-Point Iterations , 2011, SIAM J. Numer. Anal..

[23]  Euhanna Ghadimi,et al.  Optimal Parameter Selection for the Alternating Direction Method of Multipliers (ADMM): Quadratic Problems , 2013, IEEE Transactions on Automatic Control.

[24]  Hans De Sterck,et al.  On the Asymptotic Linear Convergence Speed of Anderson Acceleration, Nesterov Acceleration, and Nonlinear GMRES , 2020, SIAM J. Sci. Comput..