Distributed Convex Optimization with Many Convex Constraints

We address the problem of solving convex optimization problems with many convex constraints in a distributed setting. Our approach is based on an extension of the alternating direction method of multipliers (ADMM) that recently gained a lot of attention in the Big Data context. Although it has been invented decades ago, ADMM so far can be applied only to unconstrained problems and problems with linear equality or inequality constraints. Our extension can handle arbitrary inequality constraints directly. It combines the ability of ADMM to solve convex optimization problems in a distributed setting with the ability of the Augmented Lagrangian method to solve constrained optimization problems, and as we show, it inherits the convergence guarantees of ADMM and the Augmented Lagrangian method.

[1]  Jorge Nocedal,et al.  Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization , 1997, TOMS.

[2]  Neal E. Young,et al.  Sequential and parallel algorithms for mixed packing and covering , 2001, Proceedings 2001 IEEE International Conference on Cluster Computing.

[3]  Danny Raz,et al.  Fast, Distributed Approximation Algorithms for Positive Linear Programming with Applications to Flow Control , 2004, SIAM J. Comput..

[4]  Sören Laue,et al.  A Fast Parallel Implementation of a PTAS for Fractional Packing and Covering Linear Programs , 2015, International Journal of Parallel Programming.

[5]  Zhi-Quan Luo,et al.  On the linear convergence of the alternating direction method of multipliers , 2012, Mathematical Programming.

[6]  Alexander J. Smola,et al.  Second Order Cone Programming Approaches for Handling Missing and Uncertain Data , 2006, J. Mach. Learn. Res..

[7]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[8]  Jochen Könemann,et al.  Faster and simpler algorithms for multicommodity flow and other fractional packing problems , 1998, Proceedings 39th Annual Symposium on Foundations of Computer Science (Cat. No.98CB36280).

[9]  E. Alper Yildirim,et al.  Two Algorithms for the Minimum Enclosing Ball Problem , 2008, SIAM J. Optim..

[10]  M. J. D. Powell,et al.  Algorithms for nonlinear constraints that use lagrangian functions , 1978, Math. Program..

[11]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[12]  Jorge Nocedal,et al.  Remark on “algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound constrained optimization” , 2011, TOMS.

[13]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[14]  Zhixun Su,et al.  Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation , 2011, NIPS.

[15]  Tim Roughgarden,et al.  Fully Distributed Algorithms for Convex Optimization Problems , 2010, SIAM J. Optim..

[16]  Shiqian Ma,et al.  On the Global Linear Convergence of the ADMM with MultiBlock Variables , 2014, SIAM J. Optim..

[17]  Shiqian Ma,et al.  Sparse Inverse Covariance Selection via Alternating Linearization Methods , 2010, NIPS.

[18]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..

[19]  M. Hestenes.  Multiplier and gradient methods , 1969 .

[20]  Nikos D. Sidiropoulos,et al.  Consensus-ADMM for General Quadratically Constrained Quadratic Programming , 2016, IEEE Transactions on Signal Processing.

[21]  Nathan Srebro,et al.  SVM optimization: inverse dependence on training set size , 2008, ICML '08.

[22]  Baruch Awerbuch,et al.  A Distributed Algorithm for Large-Scale Generalized Matching , 2013, Proc. VLDB Endow..

[23]  Bernd Gärtner,et al.  Fast and Robust Smallest Enclosing Balls , 1999, ESA.

[24]  Xiangfeng Wang,et al.  Multi-Agent Distributed Optimization via Inexact Consensus ADMM , 2014, IEEE Transactions on Signal Processing.

[25]  Christos Koufogiannakis,et al.  Distributed algorithms for covering, packing and maximum weighted matching , 2011, Distributed Computing.

[26]  Euhanna Ghadimi,et al.  Optimal Parameter Selection for the Alternating Direction Method of Multipliers (ADMM): Quadratic Problems , 2013, IEEE Transactions on Automatic Control.

[27]  Sonia Martínez,et al.  On Distributed Convex Optimization Under Inequality and Equality Constraints , 2010, IEEE Transactions on Automatic Control.

[28]  James T. Kwok,et al.  Asynchronous Distributed ADMM for Consensus Optimization , 2014, ICML.

[29]  Bernd Gärtner,et al.  Fast Smallest-Enclosing-Ball Computation in High Dimensions , 2003, ESA.

[30]  Michael I. Jordan,et al.  A General Analysis of the Convergence of ADMM , 2015, ICML.

[31]  Kim-Chuan Toh,et al.  An efficient inexact symmetric Gauss–Seidel based majorized ADMM for high-dimensional convex composite conic programming , 2015, Mathematical Programming.