First-Order Algorithms for Convex Optimization with Nonseparable Objective and Coupled Constraints

In this paper, we consider a block-structured convex optimization model, where in the objective the block variables are nonseparable and they are further linearly coupled in the constraint. For the 2-block case, we propose a number of first-order algorithms to solve this model. First, the alternating direction method of multipliers (ADMM) is extended, assuming that it is easy to optimize the augmented Lagrangian function with one block of variables at each time while fixing the other block. We prove that iteration complexity bound holds under suitable conditions, where t is the number of iterations. If the subroutines of the ADMM cannot be implemented, then we propose new alternative algorithms to be called alternating proximal gradient method of multipliers, alternating gradient projection method of multipliers, and the hybrids thereof. Under suitable conditions, the iteration complexity bound is shown to hold for all the newly proposed algorithms. Finally, we extend the analysis for the ADMM to the general multi-block case.

[1]  H. H. Rachford,et al.  On the numerical solution of heat conduction problems in two and three space variables , 1956 .

[2]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[3]  Dimitri P. Bertsekas,et al.  Constrained Optimization and Lagrange Multiplier Methods , 1982 .

[4]  R. Glowinski,et al.  Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics , 1987 .

[5]  Jonathan Eckstein.  Splitting methods for monotone operators with applications to parallel optimization , 1989 .

[6]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[7]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[8]  Jukka Paatero,et al.  A model for generating household electricity load profiles , 2006 .

[9]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[10]  Wotao Yin,et al.  Bregman Iterative Algorithms for (cid:2) 1 -Minimization with Applications to Compressed Sensing ∗ , 2008 .

[11]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[12]  Jieping Ye,et al.  Large-scale sparse logistic regression , 2009, KDD.

[13]  Asuman Ozdaglar,et al.  Cooperative distributed multi-agent optimization , 2010, Convex Optimization in Signal Processing and Communications.

[14]  Shiqian Ma,et al.  Sparse Inverse Covariance Selection via Alternating Linearization Methods , 2010, NIPS.

[15]  Na Li,et al.  Optimal demand response based on utility maximization in power networks , 2011, 2011 IEEE Power and Energy Society General Meeting.

[16]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[17]  Anna Scaglione,et al.  Coordinated home energy management for real-time power balancing , 2012, 2012 IEEE Power and Energy Society General Meeting.

[18]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[19]  Xiaoming Yuan,et al.  Convergence rate and iteration complexity on the alternating direction method of multipliers with a substitution procedure for separable convex programming , 2012 .

[20]  Bingsheng He,et al.  Linearized Alternating Direction Method with Gaussian Back Substitution for Separable Convex Programming , 2011 .

[21]  Anna Scaglione,et al.  Demand-Side Management in the Smart Grid: Information Processing for the Power Switch , 2012, IEEE Signal Processing Magazine.

[22]  Daniel Boley,et al.  Local Linear Convergence of the Alternating Direction Method of Multipliers on Quadratic or Linear Programs , 2013, SIAM J. Optim..

[23]  Renato D. C. Monteiro,et al.  Iteration-Complexity of Block-Decomposition Algorithms and the Alternating Direction Method of Multipliers , 2013, SIAM J. Optim..

[24]  Wotao Yin,et al.  On the o(1/k) Convergence and Parallelization of the Alternating Direction Method of Multipliers , 2013 .

[25]  Marc Teboulle,et al.  Proximal alternating linearized minimization for nonconvex and nonsmooth problems , 2013, Mathematical Programming.

[26]  Bingsheng He,et al.  On Full Jacobian Decomposition of the Augmented Lagrangian Method for Separable Convex Programming , 2015, SIAM J. Optim..

[27]  Marc Teboulle,et al.  A simple algorithm for a class of nonsmooth convex-concave saddle-point problems , 2015, Oper. Res. Lett..

[28]  Zhi-Quan Luo,et al.  Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems , 2015, ICASSP.

[29]  Shiqian Ma,et al.  On the Global Linear Convergence of the ADMM with MultiBlock Variables , 2014, SIAM J. Optim..

[30]  Kim-Chuan Toh,et al.  On the convergence properties of a majorized ADMM for linearly constrained convex optimization problems with coupled objective functions , 2015, 1502.00098.

[31]  Shiqian Ma,et al.  Iteration Complexity Analysis of Multi-block ADMM for a Family of Convex Minimization Without Strong Convexity , 2015, Journal of Scientific Computing.

[32]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..

[33]  Bingsheng He,et al.  The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent , 2014, Mathematical Programming.

[34]  Baochun Li,et al.  An Alternating Direction Method Approach to Cloud Traffic Management , 2014 .

[35]  Wotao Yin,et al.  Parallel Multi-Block ADMM with o(1 / k) Convergence , 2013, Journal of Scientific Computing.

[36]  Zhi-Quan Luo,et al.  On the linear convergence of the alternating direction method of multipliers , 2012, Mathematical Programming.

[37]  Daniel P. Robinson,et al.  A Flexible ADMM Algorithm for Big Data Applications , 2015, J. Sci. Comput..

[38]  Shiqian Ma,et al.  An Extragradient-Based Alternating Direction Method for Convex Minimization , 2017, Found. Comput. Math..

[39]  Xiang Gao,et al.  On the Information-Adaptive Variants of the ADMM: An Iteration Complexity Perspective , 2017, Journal of Scientific Computing.