Extensions of ADMM for Separable Convex Optimization Problems with Linear Equality or Inequality Constraints

Abstract. The alternating direction method of multipliers (ADMM) proposed by Glowinski and Marrocco is a benchmark algorithm for two-block separable convex optimization problems with linear equality constraints. It has been modified, specified, and generalized from various perspectives to tackle more concrete or complicated application problems. Despite its versatility and phenomenal popularity, it remains unknown whether or not the ADMM can be extended to separable convex optimization problems with linear inequality constraints. In this paper, we lay down the foundation of how to extend the ADMM to two-block and multiple-block (more than two blocks) separable convex optimization problems with linear inequality constraints. From a high-level and methodological perspective, we propose a unified framework of algorithmic design and a roadmap for convergence analysis in the context of variational inequalities, based on which it is possible to design a series of concrete ADMM-based algorithms with provable convergence in the prediction-correction structure. The proposed algorithmic framework and roadmap for convergence analysis are eligible to various convex optimization problems with different degrees of separability, in which both linear equality and linear inequality constraints can be included. The analysis is comprehensive yet can be presented by elementary mathematics, and hence generically understandable.

[1]  Robert A. Lordo,et al.  Learning from Data: Concepts, Theory, and Methods , 2001, Technometrics.

[2]  Bingsheng He,et al.  A new inexact alternating directions method for monotone variational inequalities , 2002, Math. Program..

[3]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[4]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[5]  Jin Zhang,et al.  Discerning the Linear Convergence of ADMM for Structured Convex Optimization through the Lens of Variational Analysis , 2020, J. Mach. Learn. Res..

[6]  Georgios B. Giannakis,et al.  Consensus-Based Distributed Support Vector Machines , 2010, J. Mach. Learn. Res..

[7]  HeBingsheng,et al.  On non-ergodic convergence rate of Douglas---Rachford alternating direction method of multipliers , 2015 .

[8]  Bingsheng He,et al.  Convergence Rate Analysis for the Alternating Direction Method of Multipliers with a Substitution Procedure for Separable Convex Programming , 2017, Math. Oper. Res..

[9]  Amir Beck,et al.  First-Order Methods in Optimization , 2017 .

[10]  Bingsheng He,et al.  A class of ADMM-based algorithms for three-block separable convex programming , 2018, Computational Optimization and Applications.

[11]  Alexandre d'Aspremont,et al.  Model Selection Through Sparse Max Likelihood Estimation Model Selection Through Sparse Maximum Likelihood Estimation for Multivariate Gaussian or Binary Data , 2022 .

[12]  Bingsheng He,et al.  Linearized Alternating Direction Method with Gaussian Back Substitution for Separable Convex Programming , 2011 .

[13]  Jonathan Eckstein.  Augmented Lagrangian and Alternating Direction Methods for Convex Optimization: A Tutorial and Some Illustrative Computational Results , 2012 .

[14]  Bingsheng He,et al.  The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent , 2014, Mathematical Programming.

[15]  Antonin Chambolle,et al.  An introduction to continuous optimization for imaging , 2016, Acta Numerica.

[16]  Yuh-Jye Lee,et al.  SSVM: A Smooth Support Vector Machine for Classification , 2001, Comput. Optim. Appl..

[17]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[18]  R. Glowinski,et al.  Numerical Methods for Nonlinear Variational Problems , 1985 .

[19]  M. Powell.  A method for nonlinear constraints in minimization problems , 1969 .

[20]  R. Glowinski,et al.  Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics , 1987 .

[21]  Bingsheng He,et al.  Optimally linearizing the alternating direction method of multipliers for convex programming , 2019, Comput. Optim. Appl..

[22]  Simon Haykin,et al.  Generalized support vector machines , 1999, ESANN.

[23]  M. Fortin,et al.  Augmented Lagrangian methods : applications to the numerical solution of boundary-value problems , 1983 .

[24]  Dustin Boswell,et al.  Introduction to Support Vector Machines , 2002 .

[25]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[26]  Roland Glowinski,et al.  On Alternating Direction Methods of Multipliers: A Historical Perspective , 2014, Modeling, Simulation and Optimization for Science and Technology.

[27]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[28]  M. Hestenes.  Multiplier and gradient methods , 1969 .