The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent

The alternating direction method of multipliers (ADMM) is now widely used in many fields, and its convergence was proved when two blocks of variables are alternatively updated. It is strongly desirable and practically valuable to extend the ADMM directly to the case of a multi-block convex minimization problem where its objective function is the sum of more than two separable convex functions. However, the convergence of this extension has been missing for a long time—neither an affirmative convergence proof nor an example showing its divergence is known in the literature. In this paper we give a negative answer to this long-standing open question: The direct extension of ADMM is not necessarily convergent. We present a sufficient condition to ensure the convergence of the direct extension of ADMM, and give an example to show its divergence.

[1]  M. Powell.  A method for nonlinear constraints in minimization problems , 1969 .

[2]  M. Hestenes.  Multiplier and gradient methods , 1969 .

[3]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[4]  W. Oettli,et al.  Mathematische Optimierung : Grundlagen und Verfahren , 1975 .

[5]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[6]  R. Tyrrell Rockafellar,et al.  Augmented Lagrangians and Applications of the Proximal Point Algorithm in Convex Programming , 1976, Math. Oper. Res..

[7]  R. Glowinski,et al.  Finite element approximation and iterative solution of a class of mildly non-linear elliptic equations , 1978 .

[8]  E. G. Gol'shtein,et al.  Modified Lagrangians in Convex Programming and their Generalizations , 1979 .

[9]  Dimitri P. Bertsekas,et al.  Constrained Optimization and Lagrange Multiplier Methods , 1982 .

[10]  M. Fortin,et al.  On decomposition - coordination methods using an augmented Lagrangian , 1983 .

[11]  R. Glowinski,et al.  Numerical Methods for Nonlinear Variational Problems , 1985 .

[12]  G. McLachlan.  Discriminant Analysis and Statistical Pattern Recognition , 1992 .

[13]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[14]  M. Korda.  Grundlagen und Verfahren , 1999 .

[15]  Geoffrey J. McLachlan,et al.  Discriminant Analysis and Statistical Pattern Recognition: McLachlan/Discriminant Analysis & Pattern Recog , 2005 .

[16]  John Wright,et al.  RASL: Robust alignment by sparse and low-rank decomposition for linearly correlated images , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[17]  Pablo A. Parrilo,et al.  Latent variable graphical model selection via convex optimization , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[18]  Wotao Yin,et al.  Alternating direction augmented Lagrangian methods for semidefinite programming , 2010, Math. Program. Comput..

[19]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[20]  Xiaoming Yuan,et al.  Recovering Low-Rank and Sparse Components of Matrices from Incomplete and Noisy Observations , 2011, SIAM J. Optim..

[21]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[22]  Xiaoming Yuan,et al.  Convergence rate and iteration complexity on the alternating direction method of multipliers with a substitution procedure for separable convex programming , 2012 .

[23]  John Wright,et al.  RASL: Robust Alignment by Sparse and Low-Rank Decomposition for Linearly Correlated Images , 2012, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Xiaoming Yuan,et al.  A Note on the Alternating Direction Method of Multipliers , 2012, J. Optim. Theory Appl..

[25]  Bingsheng He,et al.  Linearized Alternating Direction Method with Gaussian Back Substitution for Separable Convex Programming , 2011 .

[26]  Jonathan Eckstein.  Augmented Lagrangian and Alternating Direction Methods for Convex Optimization: A Tutorial and Some Illustrative Computational Results , 2012 .

[27]  Su-In Lee,et al.  Node-based learning of multiple Gaussian graphical models , 2013, J. Mach. Learn. Res..

[28]  Roland Glowinski,et al.  On Alternating Direction Methods of Multipliers: A Historical Perspective , 2014, Modeling, Simulation and Optimization for Science and Technology.

[29]  Xiaoming Yuan,et al.  A splitting method for separable convex programming , 2015 .

[30]  Zhi-Quan Luo,et al.  On the linear convergence of the alternating direction method of multipliers , 2012, Mathematical Programming.