On the Optimal Linear Convergence Rate of a Generalized Proximal Point Algorithm

The proximal point algorithm (PPA) has been well studied in the literature. In particular, its linear convergence rate has been studied by Rockafellar in 1976 under certain condition. We consider a generalized PPA in the generic setting of finding a zero point of a maximal monotone operator, and show that the condition proposed by Rockafellar can also sufficiently ensure the linear convergence rate for this generalized PPA. Indeed we show that these linear convergence rates are optimal. Both the exact and inexact versions of this generalized PPA are discussed. The motivation of considering this generalized PPA is that it includes as special cases the relaxed versions of some splitting methods that are originated from PPA. Thus, linear convergence results of this generalized PPA can be used to better understand the convergence of some widely used algorithms in the literature. We focus on the particular convex minimization context and specify Rockafellar’s condition to see how to ensure the linear convergence rate for some efficient numerical schemes, including the classical augmented Lagrangian method proposed by Hensen and Powell in 1969 and its relaxed version, the original alternating direction method of multipliers (ADMM) by Glowinski and Marrocco in 1975 and its relaxed version (i.e., the generalized ADMM by Eckstein and Bertsekas in 1992). Some refined conditions weaker than existing ones are proposed in these particular contexts.

[1]  M. J. D. Powell,et al.  A method for nonlinear constraints in minimization problems , 1969 .

[2]  Muhammad Aslam Noor,et al.  On convergence criteria of generalized proximal point algorithms , 2008 .

[3]  Daniel Boley,et al.  Local Linear Convergence of the Alternating Direction Method of Multipliers on Quadratic or Linear Programs , 2013, SIAM J. Optim..

[4]  Marc Teboulle,et al.  A Logarithmic-Quadratic Proximal Method for Variational Inequalities , 1999, Comput. Optim. Appl..

[5]  H. H. Rachford,et al.  The Numerical Solution of Parabolic and Elliptic Differential Equations , 1955 .

[6]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..

[7]  E. G. Gol'shtein,et al.  Modified Lagrangians in Convex Programming and their Generalizations , 1979 .

[8]  M. Fukushima,et al.  A generalized proximal point algorithm for certain non-convex minimization problems , 1981 .

[9]  Xiaoming Yuan,et al.  A proximal point algorithm revisit on the alternating direction method of multipliers , 2013 .

[10]  Xiaoming Yuan,et al.  Local Linear Convergence of the Alternating Direction Method of Multipliers for Quadratic Programs , 2013, SIAM J. Numer. Anal..

[11]  W. Oettli,et al.  Mathematische Optimierung : Grundlagen und Verfahren , 1975 .

[12]  Osman Güer.  On the convergence of the proximal point algorithm for convex minimization , 1991 .

[13]  Dimitri P. Bertsekas,et al.  Constrained Optimization and Lagrange Multiplier Methods , 1982 .

[14]  Xiaoming Yuan,et al.  A relaxed customized proximal point algorithm for separable convex programming , 2011 .

[15]  Arkadi Nemirovski,et al.  Prox-Method with Rate of Convergence O(1/t) for Variational Inequalities with Lipschitz Continuous Monotone Operators and Smooth Convex-Concave Saddle Point Problems , 2004, SIAM J. Optim..

[16]  B. Martinet,et al.  R'egularisation d''in'equations variationnelles par approximations successives , 1970 .

[17]  G. M. Korpelevich.  The extragradient method for finding saddle points and other problems , 1976 .

[18]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[19]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[20]  D. Gabay.  Applications of the method of multipliers to variational inequalities , 1983 .

[21]  M. Hestenes.  Multiplier and gradient methods , 1969 .

[22]  Marc Teboulle,et al.  Rate of Convergence Analysis of Decomposition Methods Based on the Proximal Method of Multipliers for Convex Minimization , 2014, SIAM J. Optim..

[23]  J. Burke,et al.  A Variable Metric Proximal Point Algorithm for Monotone Operators , 1999 .

[24]  Mohamed-Jalal Fadili,et al.  Convergence rates with inexact non-expansive operators , 2014, Mathematical Programming.

[25]  R. Rockafellar.  Convex Analysis: (pms-28) , 1970 .

[26]  丸山 徹.  Convex Analysisの二,三の進展について , 1977 .

[27]  Marc Teboulle,et al.  Interior projection-like methods for monotone variational inequalities , 2005, Math. Program..

[28]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[29]  Daniel Boley,et al.  Local Linear Convergence of ADMM on Quadratic or Linear Programs , 2012 .

[30]  Osman Güler,et al.  New Proximal Point Algorithms for Convex Minimization , 1992, SIAM J. Optim..

[31]  R. Tyrrell Rockafellar,et al.  Augmented Lagrangians and Applications of the Proximal Point Algorithm in Convex Programming , 1976, Math. Oper. Res..

[32]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[33]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[34]  Wotao Yin,et al.  Faster Convergence Rates of Relaxed Peaceman-Rachford and ADMM Under Regularity Assumptions , 2014, Math. Oper. Res..

[35]  Marc Teboulle,et al.  Convergence of Proximal-Like Algorithms , 1997, SIAM J. Optim..

[36]  Hong-Kun Xu.  Iterative Algorithms for Nonlinear Operators , 2002 .

[37]  Li Shen,et al.  Linear convergence of the generalized PPA and several splitting methods for the composite inclusion problem , 2015, 1508.05156.

[38]  J. Moreau.  Proximité et dualité dans un espace hilbertien , 1965 .

[39]  Bingsheng He,et al.  Generalized alternating direction method of multipliers: new theoretical insights and applications , 2015, Math. Program. Comput..

[40]  Y. Nesterov.  A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .

[41]  H. H. Rachford,et al.  On the numerical solution of heat conduction problems in two and three space variables , 1956 .

[42]  Xiaoming Yuan,et al.  A Generalized Proximal Point Algorithm and Its Convergence Rate , 2014, SIAM J. Optim..