Generalized symmetric ADMM for separable convex optimization

The alternating direction method of multipliers (ADMM) has been proved to be effective for solving separable convex optimization subject to linear constraints. In this paper, we propose a generalized symmetric ADMM (GS-ADMM), which updates the Lagrange multiplier twice with suitable stepsizes, to solve the multi-block separable convex programming. This GS-ADMM partitions the data into two group variables so that one group consists of p block variables while the other has q block variables, where $$p \ge 1$$p≥1 and $$q \ge 1$$q≥1 are two integers. The two grouped variables are updated in a Gauss–Seidel scheme, while the variables within each group are updated in a Jacobi scheme, which would make it very attractive for a big data setting. By adding proper proximal terms to the subproblems, we specify the domain of the stepsizes to guarantee that GS-ADMM is globally convergent with a worst-case $${\mathcal {O}}(1/t)$$O(1/t) ergodic convergence rate. It turns out that our convergence domain of the stepsizes is significantly larger than other convergence domains in the literature. Hence, the GS-ADMM is more flexible and attractive on choosing and using larger stepsizes of the dual variable. Besides, two special cases of GS-ADMM, which allows using zero penalty terms, are also discussed and analyzed. Compared with several state-of-the-art methods, preliminary numerical experiments on solving a sparse matrix minimization problem in the statistical learning show that our proposed method is effective and promising.

[1]  A. Willsky,et al.  Latent variable graphical model selection via convex optimization , 2010 .

[2]  Bingsheng He,et al.  On the Proximal Jacobian Decomposition of ALM for Multiple-Block Separable Convex Minimization Problems and Its Relationship to ADMM , 2016, J. Sci. Comput..

[3]  Jianchao Bai,et al.  A New Model for Sparse and Low-Rank Matrix Decomposition , 2017 .

[4]  Bingsheng He,et al.  The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent , 2014, Mathematical Programming.

[5]  Bingsheng He,et al.  On Full Jacobian Decomposition of the Augmented Lagrangian Method for Separable Convex Programming , 2015, SIAM J. Optim..

[6]  B. Dong,et al.  Alternating projection method for sparse model updating problems , 2015 .

[7]  M. Hestenes.  Multiplier and gradient methods , 1969 .

[8]  Xiaoming Yuan,et al.  Recovering Low-Rank and Sparse Components of Matrices from Incomplete and Noisy Observations , 2011, SIAM J. Optim..

[9]  Xiaoming Yuan,et al.  A splitting method for separable convex programming , 2015 .

[10]  Bingsheng He,et al.  A Strictly Contractive Peaceman-Rachford Splitting Method for Convex Programming , 2014, SIAM J. Optim..

[11]  M. Fortin,et al.  Augmented Lagrangian methods : applications to the numerical solution of boundary-value problems , 1983 .

[12]  Bingsheng He,et al.  Convergence Study on the Symmetric Version of ADMM with Larger Step Sizes , 2016, SIAM J. Imaging Sci..

[13]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[14]  R. Glowinski,et al.  Numerical Methods for Nonlinear Variational Problems , 1985 .

[15]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[16]  Wen Song,et al.  Algorithms for solving the inverse problem associated with KAK=As+1 , 2017, J. Comput. Appl. Math..

[17]  Adam J. Rothman,et al.  Sparse permutation invariant covariance estimation , 2008, 0801.4837.

[18]  Shiqian Ma,et al.  Alternating Proximal Gradient Method for Convex Minimization , 2015, Journal of Scientific Computing.

[19]  F. Facchinei,et al.  Finite-Dimensional Variational Inequalities and Complementarity Problems , 2003 .

[20]  Deren Han,et al.  A semi-proximal-based strictly contractive Peaceman-Rachford splitting method , 2015 .

[21]  Bingsheng He,et al.  Block-wise Alternating Direction Method of Multipliers for Multiple-block Convex Programming and Beyond , 2015 .

[22]  Jianchao Bai,et al.  General parameterized proximal point algorithm with applications in statistical learning , 2019, Int. J. Comput. Math..