Linear Rate Convergence of the Alternating Direction Method of Multipliers for Convex Composite Programming

In this paper, we aim to prove the linear rate convergence of the alternating direction method of multipliers (ADMM) for solving linearly constrained convex composite optimization problems. Under a mild calmness condition, which holds automatically for convex composite piecewise linear-quadratic programming, we establish the global Q-linear rate of convergence for a general semi-proximal ADMM with the dual step-length being taken in (0, (1+51/2)/2). This semi-proximal ADMM, which covers the classic one, has the advantage to resolve the potentially nonsolvability issue of the subproblems in the classic ADMM and possesses the abilities of handling the multi-block cases efficiently. We demonstrate the usefulness of the obtained results when applied to two- and multi-block convex quadratic (semidefinite) programming.

[1]  Kim-Chuan Toh,et al.  An efficient inexact symmetric Gauss–Seidel based majorized ADMM for high-dimensional convex composite conic programming , 2015, Mathematical Programming.

[2]  Ying Cui,et al.  On the Convergence Properties of a Majorized Alternating Direction Method of Multipliers for Linearly Constrained Convex Optimization Problems with Coupled Objective Functions , 2016, J. Optim. Theory Appl..

[3]  R. Glowinski.  Lectures on Numerical Methods for Non-Linear Variational Problems , 1981 .

[4]  Paul Tseng,et al.  Hankel Matrix Rank Minimization with Applications to System Identification and Realization , 2013, SIAM J. Matrix Anal. Appl..

[5]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[6]  Kim-Chuan Toh,et al.  A note on the convergence of ADMM for linearly constrained convex optimization problems , 2015, Computational Optimization and Applications.

[7]  Kim-Chuan Toh,et al.  A Schur complement based semi-proximal ADMM for convex quadratic conic programming and extensions , 2014, Mathematical Programming.

[8]  Daniel Boley,et al.  Local Linear Convergence of ADMM on Quadratic or Linear Programs , 2012 .

[9]  F. Luque.  Asymptotic convergence analysis of the proximal point algorithm , 1984 .

[10]  Jie Sun.  On monotropic piecewise quadratic programming (network, algorithm, convex programming, decomposition method) , 1986 .

[11]  Daniel Boley,et al.  Local Linear Convergence of the Alternating Direction Method of Multipliers on Quadratic or Linear Programs , 2013, SIAM J. Optim..

[12]  R. Tyrrell Rockafellar,et al.  Augmented Lagrangians and Applications of the Proximal Point Algorithm in Convex Programming , 1976, Math. Oper. Res..

[13]  Jonathan Eckstein.  Some Saddle-function splitting methods for convex programming , 1994 .

[14]  S. M. Robinson.  Some continuity properties of polyhedral multifunctions , 1981 .

[15]  Bastian Goldlücke,et al.  Variational Analysis , 2014, Computer Vision, A Reference Guide.

[16]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[17]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[18]  Defeng Sun,et al.  Characterization of the Robust Isolated Calmness for a Class of Conic Programming Problems , 2017, SIAM J. Optim..

[19]  Jonathan Eckstein.  Splitting methods for monotone operators with applications to parallel optimization , 1989 .

[20]  Marc Teboulle,et al.  Rate of Convergence Analysis of Decomposition Methods Based on the Proximal Method of Multipliers for Convex Minimization , 2014, SIAM J. Optim..

[21]  Kim-Chuan Toh,et al.  On the convergence properties of a majorized ADMM for linearly constrained convex optimization problems with coupled objective functions , 2015, 1502.00098.

[22]  Roland Glowinski,et al.  On Alternating Direction Methods of Multipliers: A Historical Perspective , 2014, Modeling, Simulation and Optimization for Science and Technology.

[23]  Xiaoming Yuan,et al.  Local Linear Convergence of the Alternating Direction Method of Multipliers for Quadratic Programs , 2013, SIAM J. Numer. Anal..

[24]  S. M. Robinson.  Generalized equations and their solutions, Part I: Basic theory , 1979 .

[25]  Bingsheng He,et al.  A new inexact alternating directions method for monotone variational inequalities , 2002, Math. Program..

[26]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[27]  Wei Hong Yang,et al.  Linear Convergence of the Alternating Direction Method of Multipliers for a Class of Convex Optimization Problems , 2016, SIAM J. Numer. Anal..

[29]  O. Mangasarian,et al.  Local Duality of Nonlinear Programs , 1984 .

[30]  Zhi-Quan Luo,et al.  On the linear convergence of the alternating direction method of multipliers , 2012, Mathematical Programming.

[31]  K. Toh,et al.  On the Asymptotic Superlinear Convergence of the Augmented Lagrangian Method for Semidefinite Programming with Multiple Solutions , 2016, 1610.00875.

[32]  Houduo Qi,et al.  Local Duality of Nonlinear Semidefinite Programming , 2009, Math. Oper. Res..

[33]  T. Wu,et al.  A Class of Linearized Proximal Alternating Direction Methods , 2011, J. Optim. Theory Appl..

[34]  M. Fortin,et al.  On decomposition - coordination methods using an augmented Lagrangian , 1983 .

[35]  J. Frédéric Bonnans,et al.  Perturbation Analysis of Optimization Problems , 2000, Springer Series in Operations Research.

[36]  Bingsheng He,et al.  The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent , 2014, Mathematical Programming.

[37]  S. M. Robinson.  An Implicit-Function Theorem for Generalized Variational Inequalities. , 1976 .

[38]  R. Rockafellar,et al.  Implicit Functions and Solution Mappings , 2009 .

[39]  Kim-Chuan Toh,et al.  A Majorized ADMM with Indefinite Proximal Terms for Linearly Constrained Convex Composite Optimization , 2014, SIAM J. Optim..

[40]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..

[41]  丸山 徹.  Convex Analysisの二,三の進展について , 1977 .

[42]  Kim-Chuan Toh,et al.  A Convergent 3-Block SemiProximal Alternating Direction Method of Multipliers for Conic Programming with 4-Type Constraints , 2014, SIAM J. Optim..

[43]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[44]  Jonathan Eckstein,et al.  Understanding the Convergence of the Alternating Direction Method of Multipliers: Theoretical and Computational Perspectives , 2015 .

[45]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[46]  D. Gabay.  Applications of the method of multipliers to variational inequalities , 1983 .