On Glowinski’s Open Question on the Alternating Direction Method of Multipliers

The alternating direction method of multipliers was proposed by Glowinski and Marrocco in 1974, and it has been widely used in a broad spectrum of areas, especially in some sparsity-driven application domains. In 1982, Fortin and Glowinski suggested to enlarge the range of the dual step size for updating the multiplier from 1 to the open interval of zero to the golden ratio, and this strategy immediately accelerates the convergence of alternating direction method of multipliers for most of its applications. Meanwhile, Glowinski raised the question of whether or not the range can be further enlarged to the open interval of zero to 2; this question remains open with nearly no progress in the past decades. In this paper, we answer this question affirmatively for the case where both the functions in the objective function are quadratic. Thus, Glowinski’s open question is partially answered. We further establish the global linear convergence of the alternating direction method of multipliers with this enlarged step size range for the quadratic programming under a tight condition.

[1]  Leon O. Chua,et al.  Linear and nonlinear circuits , 1987 .

[2]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[3]  M. Fortin,et al.  On decomposition - coordination methods using an augmented Lagrangian , 1983 .

[4]  Jonathan Eckstein.  Some Saddle-function splitting methods for convex programming , 1994 .

[5]  A. Stuart,et al.  Portfolio Selection: Efficient Diversification of Investments , 1959 .

[6]  Bingsheng He,et al.  Convergence Study on the Symmetric Version of ADMM with Larger Step Sizes , 2016, SIAM J. Imaging Sci..

[7]  Bingsheng He,et al.  Some convergence properties of a method of multipliers for linearly constrained monotone variational inequalities , 1998, Oper. Res. Lett..

[8]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[9]  M. H. Xu.  Proximal Alternating Directions Method for Structured Variational Inequalities , 2007 .

[10]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[11]  Su Zhang,et al.  A modified alternating direction method for convex quadratically constrained quadratic semidefinite programs , 2010, Eur. J. Oper. Res..

[12]  A. N. Tikhonov,et al.  Solutions of ill-posed problems , 1977 .

[13]  M. Fortin,et al.  Augmented Lagrangian methods : applications to the numerical solution of boundary-value problems , 1983 .

[14]  R. Tyrrell Rockafellar,et al.  Augmented Lagrangians and Applications of the Proximal Point Algorithm in Convex Programming , 1976, Math. Oper. Res..

[15]  Jonas Schmitt.  Portfolio Selection Efficient Diversification Of Investments , 2016 .

[16]  Daniel Boley,et al.  Local Linear Convergence of the Alternating Direction Method of Multipliers on Quadratic or Linear Programs , 2013, SIAM J. Optim..

[17]  Xiaoming Yuan,et al.  On the O(1/t) Convergence Rate of Alternating Direction Method with Logarithmic-Quadratic Proximal Regularization , 2012, SIAM J. Optim..

[18]  M. Fiedler.  Bounds for the determinant of the sum of hermitian matrices , 1971 .

[19]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[20]  E. G. Gol'shtein,et al.  Modified Lagrangians in Convex Programming and their Generalizations , 1979 .

[21]  Wotao Yin,et al.  Alternating direction augmented Lagrangian methods for semidefinite programming , 2010, Math. Program. Comput..

[22]  R. Glowinski.  Lectures on Numerical Methods for Non-Linear Variational Problems , 1981 .

[23]  M. J. D. Powell,et al.  A method for nonlinear constraints in minimization problems , 1969 .

[25]  Noah H. Rhee,et al.  On the equality of algebraic and geometric multiplicities of matrix eigenvalues , 2011, Appl. Math. Lett..

[26]  J. Navarro-Pedreño.  Numerical Methods for Least Squares Problems , 1996 .

[27]  Caihua Chen,et al.  Extended ADMM and BCD for nonseparable convex minimization models with quadratic coupling terms: convergence analysis and insights , 2015, Mathematical Programming.

[28]  D. Gabay.  Applications of the method of multipliers to variational inequalities , 1983 .

[29]  Bingsheng He,et al.  A new inexact alternating directions method for monotone variational inequalities , 2002, Math. Program..

[30]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[31]  Arthur R. Bergen,et al.  Power Systems Analysis , 1986 .

[32]  Min Tao,et al.  Rigorous convergence analysis of alternating variable minimization with multiplier methods for quadratic programming problems with equality constraints , 2016 .

[33]  Roland Glowinski,et al.  On Alternating Direction Methods of Multipliers: A Historical Perspective , 2014, Modeling, Simulation and Optimization for Science and Technology.

[34]  Xiaoming Yuan,et al.  Local Linear Convergence of the Alternating Direction Method of Multipliers for Quadratic Programs , 2013, SIAM J. Numer. Anal..

[35]  R. Bhatia.  Matrix Analysis , 1996 .

[36]  M. Hestenes.  Multiplier and gradient methods , 1969 .

[37]  Xiaoming Yuan,et al.  The generalized proximal point algorithm with step size 2 is not necessarily convergent , 2018, Comput. Optim. Appl..

[38]  Jonathan Eckstein.  Augmented Lagrangian and Alternating Direction Methods for Convex Optimization: A Tutorial and Some Illustrative Computational Results , 2012 .

[39]  C. Eden.  BookOn systems analysis : David Berlinski 186 pages, £ 10.25 (Cambridge, Mass, and London, MIT Press, 1976)☆ , 1978 .

[40]  Matthias Ehrgott,et al.  Interactive decision support in radiation therapy treatment planning , 2011, OR Spectr..

[41]  Carl D. Meyer,et al.  Matrix Analysis and Applied Linear Algebra , 2000 .

[42]  Masao Fukushima,et al.  Some Reformulations and Applications of the Alternating Direction Method of Multipliers , 1994 .