A General Analysis of the Convergence of ADMM

We provide a new proof of the linear convergence of the alternating direction method of multipliers (ADMM) when one of the objective terms is strongly convex. Our proof is based on a framework for analyzing optimization algorithms introduced in Lessard et al. (2014), reducing algorithm convergence to verifying the stability of a dynamical system. This approach generalizes a number of existing results and obviates any assumptions about specific choices of algorithm parameters. On a numerical example, we demonstrate that minimizing the derived bound on the convergence rate provides a practical approach to selecting algorithm parameters for particular ADMM instances. We complement our upper bound by constructing a nearly-matching lower bound on the worst-case rate of convergence.

[1]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[2]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[3]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[4]  Numérisation de documents anciens mathématiques,et al.  Mathematical modelling and numerical analysis : Modélisation mathématique et analyse numérique. , 1985 .

[5]  M. Corelss.  Guaranteed rates of exponential convergence for uncertain systems , 1990 .

[6]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[7]  Jonathan Eckstein.  Parallel alternating direction multiplier decomposition of convex programs , 1994 .

[8]  Michael C. Ferris,et al.  Operator-Splitting Methods for Monotone Affine Variational Inequalities, with a Parallel Application to Optimal Control , 1998, INFORMS J. Comput..

[9]  José M. Bioucas-Dias,et al.  Alternating direction algorithms for constrained sparse regression: Application to hyperspectral unmixing , 2010, 2010 2nd Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing.

[10]  Georgios B. Giannakis,et al.  Consensus-Based Distributed Support Vector Machines , 2010, J. Mach. Learn. Res..

[11]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[12]  Ofer Meshi,et al.  An Alternating Direction Method for Dual MAP LP Relaxation , 2011, ECML/PKDD.

[13]  Arindam Banerjee,et al.  Online Alternating Direction Method , 2012, ICML.

[14]  Stephen P. Boyd,et al.  An ADMM Algorithm for a Class of Total Variation Regularized Estimation Problems , 2012, 1203.1828.

[15]  Kang G. Shin,et al.  Efficient Distributed Linear Classification Algorithms via the Alternating Direction Method of Multipliers , 2012, AISTATS.

[16]  Massimiliano Pontil,et al.  A New Convex Relaxation for Tensor Completion , 2013, NIPS.

[17]  Xinhua Zhang,et al.  Convex Two-Layer Modeling , 2013, NIPS.

[18]  Pradeep Ravikumar,et al.  Large Scale Distributed Sparse Precision Estimation , 2013, NIPS.

[19]  Alexander T. Ihler,et al.  Linear Approximation to ADMM for MAP inference , 2013, ACML.

[20]  M. Corless,et al.  Incremental quadratic stability , 2013 .

[21]  A. Bemporad,et al.  Forward-backward truncated Newton methods for convex composite optimization , 2014, 1402.6655.

[22]  Alexander J. Smola,et al.  Communication Efficient Distributed Machine Learning with the Parameter Server , 2014, NIPS.

[23]  Alberto Bemporad,et al.  Douglas-rachford splitting: Complexity estimates and accelerated variants , 2014, 53rd IEEE Conference on Decision and Control.

[24]  Sarah Bird,et al.  Optimizing Resource Allocations for Dynamic Interactive Applications , 2014 .

[25]  A. Bemporad,et al.  Forward-backward truncated Newton methods for large-scale convex composite optimization , 2014 .

[26]  Stephen P. Boyd,et al.  Diagonal scaling in Douglas-Rachford splitting and ADMM , 2014, 53rd IEEE Conference on Decision and Control.

[27]  Guillaume Bouchard,et al.  Overlapping Trace Norms in Multi-View Learning , 2014, ArXiv.

[28]  Pascal Bianchi,et al.  Linear convergence rate for distributed optimization with the alternating direction method of multipliers , 2014, 53rd IEEE Conference on Decision and Control.

[29]  Anima Anandkumar,et al.  Multi-Step Stochastic ADMM in High Dimensions: Applications to Sparse Optimization and Matrix Decomposition , 2014, NIPS.

[30]  James T. Kwok,et al.  Asynchronous Distributed ADMM for Consensus Optimization , 2014, ICML.

[31]  Richard G. Baraniuk,et al.  Fast Alternating Direction Optimization Methods , 2014, SIAM J. Imaging Sci..

[32]  Euhanna Ghadimi,et al.  Optimal Parameter Selection for the Alternating Direction Method of Multipliers (ADMM): Quadratic Problems , 2013, IEEE Transactions on Automatic Control.

[33]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..

[34]  Benjamin Recht,et al.  Analysis and Design of Optimization Algorithms via Integral Quadratic Constraints , 2014, SIAM J. Optim..

[35]  Damek Davis,et al.  Convergence Rate Analysis of Several Splitting Schemes , 2014, 1406.4834.

[36]  Wotao Yin,et al.  Faster Convergence Rates of Relaxed Peaceman-Rachford and ADMM Under Regularity Assumptions , 2014, Math. Oper. Res..

[37]  Zhi-Quan Luo,et al.  On the linear convergence of the alternating direction method of multipliers , 2012, Mathematical Programming.