Local Linear Convergence of the Alternating Direction Method of Multipliers on Quadratic or Linear Programs

We introduce a novel matrix recurrence yielding a new spectral analysis of the local transient convergence behavior of the alternating direction method of multipliers (ADMM), for the particular case of a quadratic program or a linear program. We identify a particular combination of vector iterates whose convergence can be analyzed via a spectral analysis. The theory predicts that ADMM should go through up to four convergence regimes, such as constant step convergence or linear convergence, ending with the latter when close enough to the optimal solution if the optimal solution is unique and satisfies strict complementarity.

[1]  H. H. Rachford,et al.  On the numerical solution of heat conduction problems in two and three space variables , 1956 .

[2]  R. Glowinski,et al.  Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics , 1987 .

[3]  Radhakrishnan Mahadevan,et al.  Economics of membrane occupancy and respiro-fermentation , 2011, Molecular systems biology.

[4]  Benar Fux Svaiter,et al.  General Projective Splitting Methods for Sums of Maximal Monotone Operators , 2009, SIAM J. Control. Optim..

[5]  Bingsheng He,et al.  A new inexact alternating directions method for monotone variational inequalities , 2002, Math. Program..

[6]  Y. Nesterov.  A method for unconstrained convex minimization problem with the rate of convergence o(1/k^2) , 1983 .

[7]  Paul Tseng,et al.  A Modified Forward-backward Splitting Method for Maximal Monotone Mappings 1 , 1998 .

[8]  Shiqian Ma,et al.  Fast alternating linearization methods for minimizing the sum of two convex functions , 2009, Math. Program..

[9]  D. Sorensen.  Numerical methods for large eigenvalue problems , 2002, Acta Numerica.

[10]  Bingsheng He,et al.  On non-ergodic convergence rate of Douglas–Rachford alternating direction method of multipliers , 2014, Numerische Mathematik.

[11]  Peter Lancaster,et al.  The theory of matrices , 1969 .

[12]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[13]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[14]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[15]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[16]  M. Fortin,et al.  Augmented Lagrangian methods : applications to the numerical solution of boundary-value problems , 1983 .

[17]  João M. F. Xavier,et al.  Distributed Basis Pursuit , 2010, IEEE Transactions on Signal Processing.

[18]  J. Ortega.  Numerical Analysis: A Second Course , 1974 .

[19]  F. R. Gantmakher.  The Theory of Matrices , 1984 .

[20]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[21]  Michael K. Ng,et al.  Solving Constrained Total-variation Image Restoration and Reconstruction Problems via Alternating Direction Methods , 2010, SIAM J. Sci. Comput..

[22]  Richard G. Baraniuk,et al.  Fast Alternating Direction Optimization Methods , 2014, SIAM J. Imaging Sci..

[23]  Emiliano Dall'Anese,et al.  Fast Consensus by the Alternating Direction Multipliers Method , 2011, IEEE Transactions on Signal Processing.

[24]  D. Donoho,et al.  Basis pursuit , 1994, Proceedings of 1994 28th Asilomar Conference on Signals, Systems and Computers.

[25]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[26]  R. Tibshirani,et al.  Sparsity and smoothness via the fused lasso , 2005 .

[27]  Y. Nesterov.  Gradient methods for minimizing composite objective function , 2007 .

[28]  Arindam Banerjee,et al.  Online Alternating Direction Method , 2012, ICML.

[29]  Junfeng Yang,et al.  Alternating Direction Algorithms for 1-Problems in Compressive Sensing , 2009, SIAM J. Sci. Comput..

[30]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[31]  Chao Yang,et al.  ARPACK users' guide - solution of large-scale eigenvalue problems with implicitly restarted Arnoldi methods , 1998, Software, environments, tools.

[32]  Xiaoming Yuan,et al.  Recovering Low-Rank and Sparse Components of Matrices from Incomplete and Noisy Observations , 2011, SIAM J. Optim..

[33]  Shiqian Ma,et al.  Fast Multiple-Splitting Algorithms for Convex Optimization , 2009, SIAM J. Optim..

[34]  Georgios B. Giannakis,et al.  Group sparse Lasso for cognitive network sensing robust to model uncertainties and outliers , 2012, Phys. Commun..

[35]  João M. F. Xavier,et al.  D-ADMM: A Communication-Efficient Distributed Algorithm for Separable Optimization , 2012, IEEE Transactions on Signal Processing.

[36]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[37]  J. Douglas Faires,et al.  Numerical Analysis , 1981 .

[38]  Robert R. Meyer,et al.  A variable-penalty alternating directions method for convex optimization , 1998, Math. Program..

[39]  Xiaoming Yuan,et al.  A descent method for structured monotone variational inequalities , 2007, Optim. Methods Softw..

[40]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[41]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[42]  Bingsheng He,et al.  Solving Large-Scale Least Squares Semidefinite Programming by Alternating Direction Methods , 2011, SIAM J. Matrix Anal. Appl..

[43]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[44]  Xiaoming Yuan,et al.  Matrix completion via an alternating direction method , 2012 .

[45]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[46]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[47]  Alston S. Householder,et al.  The Theory of Matrices in Numerical Analysis , 1964 .

[48]  D. Gabay.  Applications of the method of multipliers to variational inequalities , 1983 .

[49]  Masao Fukushima,et al.  Application of the alternating direction method of multipliers to separable convex programming problems , 1992, Comput. Optim. Appl..

[50]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[51]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[52]  Su Zhang,et al.  A modified alternating direction method for convex quadratically constrained quadratic semidefinite programs , 2010, Eur. J. Oper. Res..

[53]  L. Mirsky,et al.  The Theory of Matrices , 1961, The Mathematical Gazette.

[54]  Ernie Esser,et al.  Applications of Lagrangian-Based Alternating Direction Methods and Connections to Split Bregman , 2009 .

[55]  D. Bertsekas,et al.  An Alternating Direction Method for Linear Programming , 1990 .

[56]  J. Spingarn.  Partial inverse of a monotone operator , 1983 .