The augmented Lagrangian method with full Jacobian decomposition and logarithmic-quadratic proximal regularization for multiple-block separable convex programming

We consider a separable convex minimization model whose variables are coupled by linear constraints and they are subject to the positive orthant constraints, and its objective function is in form of m functions without coupled variables. It is well recognized that when the augmented Lagrangian method (ALM) is applied to solve some concrete applications, the resulting subproblem at each iteration should be decomposed to generate solvable subproblems. When the Gauss-Seidel decomposition is implemented, this idea has inspired the alternating direction method of multiplier (form = 2) and its variants (form ≥ 3). When the Jacobian decomposition is considered, it has been shown that the ALM with Jacobian decomposition in its subproblem is not necessarily convergent even when m = 2 and it was suggested to regularize the decomposed subproblems with quadratic proximal terms to ensure the convergence. In this paper, we focus on the multiple-block case with m ≥ 3. We consider implementing the full Jacobian decomposition to ALM’s subproblems and using the logarithmic-quadratic proximal (LQP) terms to regularize the decomposed subproblems. The resulting subproblems are all unconstrained minimization problems because the positive orthant constraints are all inactive; and they are fully eligible for parallel computation. Accordingly, the ALM with full Jacobian decomposition and LQP regularization is proposed. We also consider its inexact version which allows the subproblems to be solved inexactly. For both the exact and inexact versions, we comprehensively discuss their convergence, including their global convergence, worst-case convergence rates measured by the iteration-complexity in both the ergodic and nonergodic senses, and linear convergence rates under additional assumptions. Some preliminary numerical results are reported to demonstrate the efficiency of the ALM with full Jacobian decomposition and LQP regularization. 2010 Mathematics Subject Classification. 90C25, 90C33, 65K05.

[1]  Jefferson G. Melo,et al.  Iteration-complexity of a Jacobi-type non-Euclidean ADMM for multi-block linearly constrained nonconvex programs , 2017, 1705.07229.

[2]  Wotao Yin,et al.  Parallel Multi-Block ADMM with o(1 / k) Convergence , 2013, Journal of Scientific Computing.

[3]  Zhi-Quan Luo,et al.  On the linear convergence of the alternating direction method of multipliers , 2012, Mathematical Programming.

[4]  Bingsheng He,et al.  On the Proximal Jacobian Decomposition of ALM for Multiple-Block Separable Convex Minimization Problems and Its Relationship to ADMM , 2016, J. Sci. Comput..

[5]  Kim-Chuan Toh,et al.  A Majorized ADMM with Indefinite Proximal Terms for Linearly Constrained Convex Composite Optimization , 2014, SIAM J. Optim..

[6]  Damek Davis,et al.  Convergence Rate Analysis of Several Splitting Schemes , 2014, 1406.4834.

[7]  Bingsheng He,et al.  The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent , 2014, Mathematical Programming.

[8]  Bingsheng He,et al.  On Full Jacobian Decomposition of the Augmented Lagrangian Method for Separable Convex Programming , 2015, SIAM J. Optim..

[9]  Caihua Chen,et al.  Further Study on the Convergence Rate of Alternating Direction Method of Multipliers with Logarithmic-quadratic Proximal Regularization , 2015, J. Optim. Theory Appl..

[10]  Bingsheng He,et al.  On non-ergodic convergence rate of Douglas–Rachford alternating direction method of multipliers , 2014, Numerische Mathematik.

[11]  Min Li,et al.  A Strictly Contractive Peaceman-Rachford Splitting Method with Logarithmic-Quadratic Proximal Regularization for Convex Programming , 2015, Math. Oper. Res..

[12]  Shiqian Ma,et al.  On the Global Linear Convergence of the ADMM with MultiBlock Variables , 2014, SIAM J. Optim..

[13]  Tianyi Lin,et al.  On the Convergence Rate of Multi-Block ADMM , 2014, 1408.4265.

[14]  Bingsheng He,et al.  A Strictly Contractive Peaceman-Rachford Splitting Method for Convex Programming , 2014, SIAM J. Optim..

[15]  Roland Glowinski,et al.  On Alternating Direction Methods of Multipliers: A Historical Perspective , 2014, Modeling, Simulation and Optimization for Science and Technology.

[16]  Min Li,et al.  Inexact Alternating Direction Methods of Multipliers with Logarithmic–Quadratic Proximal Regularization , 2013, J. Optim. Theory Appl..

[17]  Xiaoming Yuan,et al.  On the O(1/t) Convergence Rate of Alternating Direction Method with Logarithmic-Quadratic Proximal Regularization , 2012, SIAM J. Optim..

[18]  Bingsheng He,et al.  Linearized Alternating Direction Method with Gaussian Back Substitution for Separable Convex Programming , 2011 .

[19]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[20]  Jonathan Eckstein.  Augmented Lagrangian and Alternating Direction Methods for Convex Optimization: A Tutorial and Some Illustrative Computational Results , 2012 .

[21]  Min Li,et al.  An LQP-Based Decomposition Method for Solving a Class of Variational Inequalities , 2011, SIAM J. Optim..

[22]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[23]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[24]  Xiaoming Yuan,et al.  Recovering Low-Rank and Sparse Components of Matrices from Incomplete and Noisy Observations , 2011, SIAM J. Optim..

[25]  John Wright,et al.  RASL: Robust alignment by sparse and low-rank decomposition for linearly correlated images , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[26]  Michael Patriksson,et al.  A survey on the continuous nonlinear resource allocation problem , 2008, Eur. J. Oper. Res..

[27]  Y. Nesterov.  Gradient methods for minimizing composite objective function , 2007 .

[28]  Jianhong Shen,et al.  Deblurring images: Matrices, spectra, and filtering , 2007, Math. Comput..

[29]  Marc Teboulle,et al.  Interior projection-like methods for monotone variational inequalities , 2005, Math. Program..

[30]  F. Facchinei,et al.  Finite-Dimensional Variational Inequalities and Complementarity Problems , 2003 .

[31]  Marc Teboulle,et al.  Entropic proximal decomposition methods for convex programs and variational inequalities , 2001, Math. Program..

[32]  Marc Teboulle,et al.  A Logarithmic-Quadratic Proximal Method for Variational Inequalities , 1999, Comput. Optim. Appl..

[33]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[34]  R. Glowinski,et al.  Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics , 1987 .

[35]  R. Glowinski,et al.  Numerical Methods for Nonlinear Variational Problems , 1985 .

[36]  Y. Nesterov.  A method for unconstrained convex minimization problem with the rate of convergence o(1/k^2) , 1983 .

[37]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .

[38]  Dimitri P. Bertsekas,et al.  Constrained Optimization and Lagrange Multiplier Methods , 1982 .

[39]  E. G. Gol'shtein,et al.  Modified Lagrangians in Convex Programming and their Generalizations , 1979 .

[40]  R. Tyrrell Rockafellar,et al.  Augmented Lagrangians and Applications of the Proximal Point Algorithm in Convex Programming , 1976, Math. Oper. Res..

[41]  B. Curtis Eaves,et al.  On the basic theorem of complementarity , 1971, Math. Program..

[42]  B. Martinet,et al.  R'egularisation d''in'equations variationnelles par approximations successives , 1970 .

[43]  M. Powell.  A method for nonlinear constraints in minimization problems , 1969 .

[44]  H. Uzawa.  Market Mechanisms and Mathematical Programming , 1960 .