On inexact ADMMs with relative error criteria

In this paper, we develop two inexact alternating direction methods of multipliers (ADMMs) with relative error criteria for which only a few parameters are needed to control the error tolerance. In many practical applications, the numerical performance is often improved if a larger step-length is used. Hence in this paper we also consider to seek a larger step-length to update the Lagrangian multiplier for better numerical efficiency. Specifically, if we only allow one subproblem in the classic ADMM to be solved inexactly by a certain relative error criterion, then a larger step-length can be used to update the Lagrangian multiplier. Related convergence analysis of those proposed algorithms is also established under the assumption that the solution set to the KKT system of the problem is not empty. Numerical experiments on solving total variation (TV)-based image denosing and analysis sparse recovery problems are provided to demonstrate the effectiveness of the proposed methods and the advantage of taking a larger step-length.

[1]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[2]  Benar Fux Svaiter,et al.  An Inexact Hybrid Generalized Proximal Point Algorithm and Some New Results on the Theory of Bregman Functions , 2000, Math. Oper. Res..

[3]  W. Hager,et al.  Inexact alternating direction multiplier methods for separable convex optimization , 2016, 1604.02494.

[4]  M. Solodov,et al.  A Hybrid Approximate Extragradient – Proximal Point Algorithm Using the Enlargement of a Maximal Monotone Operator , 1999 .

[5]  T. Wu,et al.  A Class of Linearized Proximal Alternating Direction Methods , 2011, J. Optim. Theory Appl..

[6]  Wang Yao,et al.  Approximate ADMM algorithms derived from Lagrangian splitting , 2017, Comput. Optim. Appl..

[7]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[8]  M. H. Xu.  Proximal Alternating Directions Method for Structured Variational Inequalities , 2007 .

[9]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[10]  Paulo J. S. Silva,et al.  A practical relative error criterion for augmented Lagrangians , 2012, Mathematical Programming.

[11]  Xiaobo Yang,et al.  An inexact alternating direction method of multipliers with relative error criteria , 2017, Optim. Lett..

[12]  Gene H. Golub,et al.  Matrix computations , 1983 .

[13]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[14]  Roland Glowinski,et al.  On Alternating Direction Methods of Multipliers: A Historical Perspective , 2014, Modeling, Simulation and Optimization for Science and Technology.

[15]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[16]  R. Rockafellar.  Convex Analysis: (pms-28) , 1970 .

[17]  Timothy A. Davis,et al.  Direct methods for sparse linear systems , 2006, Fundamentals of algorithms.

[18]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[19]  Paul Tseng,et al.  Hankel Matrix Rank Minimization with Applications to System Identification and Realization , 2013, SIAM J. Matrix Anal. Appl..

[20]  Kim-Chuan Toh,et al.  A note on the convergence of ADMM for linearly constrained convex optimization problems , 2015, Computational Optimization and Applications.

[21]  Bingsheng He,et al.  A new inexact alternating directions method for monotone variational inequalities , 2002, Math. Program..

[22]  Wotao Yin,et al.  Alternating direction augmented Lagrangian methods for semidefinite programming , 2010, Math. Program. Comput..

[23]  Kim-Chuan Toh,et al.  On the equivalence of inexact proximal ALM and ADMM for a class of convex composite programming , 2018, Math. Program..

[24]  Michael K. Ng,et al.  Inexact Alternating Direction Methods for Image Recovery , 2011, SIAM J. Sci. Comput..

[25]  Jonathan Eckstein.  Augmented Lagrangian and Alternating Direction Methods for Convex Optimization: A Tutorial and Some Illustrative Computational Results , 2012 .

[26]  R. Glowinski,et al.  Numerical Methods for Nonlinear Variational Problems , 1985 .

[27]  Michael Elad,et al.  The Cosparse Analysis Model and Algorithms , 2011, ArXiv.

[28]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[29]  Kim-Chuan Toh,et al.  An efficient inexact symmetric Gauss–Seidel based majorized ADMM for high-dimensional convex composite conic programming , 2015, Mathematical Programming.

[30]  Min Li,et al.  Inexact Alternating Direction Methods of Multipliers with Logarithmic–Quadratic Proximal Regularization , 2013, J. Optim. Theory Appl..

[31]  Yonina C. Eldar,et al.  Smoothing and Decomposition for Analysis Sparse Recovery , 2013, IEEE Transactions on Signal Processing.

[32]  Wang Yao,et al.  Relative-error approximate versions of Douglas–Rachford splitting and special cases of the ADMM , 2018, Math. Program..