Accelerated Uzawa methods for convex optimization

We focus on a linearly constrained strongly convex minimization model and discuss the application of the classical Uzawa method. Our principal goal is to show that some existing acceleration schemes can be used to accelerate the Uzawa method in the sense that the worst-case convergence rate (measured by the iteration complexity) of the resulting accelerated Uzawa schemes is O(1/k2) where k represents the iteration counter. Our discussion assumes that the objective function is given by a black-box oracle; thus an inexact version of the Uzawa method with a sequence of dynamically-chosen step sizes is implemented. A worst-case convergence rate of O(1/k) is also shown for this inexact version. Some preliminary numerical results are reported to verify the acceleration effectiveness of the accelerated Uzawa schemes and their superiority over some existing methods.

[1]  Jian-Feng Cai,et al.  Linearized Bregman iterations for compressed sensing , 2009, Math. Comput..

[2]  R. Glowinski,et al.  Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics , 1987 .

[3]  Raymond H. Chan,et al.  Journal of Computational and Applied Mathematics a Reduced Newton Method for Constrained Linear Least-squares Problems , 2022 .

[4]  James H. Bramble,et al.  The Lagrange multiplier method for Dirichlet’s problem , 1981 .

[5]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[6]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[7]  Stanley Osher,et al.  A Unified Primal-Dual Algorithm Framework Based on Bregman Iteration , 2010, J. Sci. Comput..

[8]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .

[9]  Wotao Yin,et al.  Analysis and Generalizations of the Linearized Bregman Method , 2010, SIAM J. Imaging Sci..

[10]  D. Bertsekas.  On the Goldstein-Levitin-Polyak gradient projection method , 1974, CDC 1974.

[11]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[12]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[13]  Hui Zhang,et al.  Strongly Convex Programming for Exact Matrix Completion and Robust Principal Component Analysis , 2011, ArXiv.

[14]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[15]  B. Martinet.  Brève communication. Régularisation d'inéquations variationnelles par approximations successives , 1970 .

[16]  Mingqiang Zhu,et al.  An Efficient Primal-Dual Hybrid Gradient Algorithm For Total Variation Image Restoration , 2008 .

[17]  Yurii Nesterov,et al.  Gradient methods for minimizing composite functions , 2012, Mathematical Programming.

[18]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[19]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[20]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[21]  Marc Teboulle,et al.  Interior Gradient and Proximal Methods for Convex and Conic Optimization , 2006, SIAM J. Optim..

[22]  Tom Goldstein,et al.  The Split Bregman Method for L1-Regularized Problems , 2009, SIAM J. Imaging Sci..

[23]  M. Hestenes.  Multiplier and gradient methods , 1969 .

[24]  Jian-Feng Cai,et al.  Linearized Bregman Iterations for Frame-Based Image Deblurring , 2009, SIAM J. Imaging Sci..

[25]  Raymond H. Chan,et al.  Constrained Total Variation Deblurring Models and Fast Algorithms Based on Alternating Direction Method of Multipliers , 2013, SIAM J. Imaging Sci..

[26]  Michel Fortin,et al.  Mixed and Hybrid Finite Element Methods , 2011, Springer Series in Computational Mathematics.

[27]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[28]  Rachid Deriche,et al.  Regularizing Flows for Constrained Matrix-Valued Images , 2004 .

[29]  ANTONIN CHAMBOLLE,et al.  An Algorithm for Total Variation Minimization and Applications , 2004, Journal of Mathematical Imaging and Vision.

[30]  B. V. Dean,et al.  Studies in Linear and Non-Linear Programming. , 1959 .

[31]  Bin Dong,et al.  Fast Linearized Bregman Iteration for Compressive Sensing and Sparse Denoising , 2011, ArXiv.

[32]  Raymond H. Chan,et al.  Conjugate Gradient Methods for Toeplitz Systems , 1996, SIAM Rev..

[33]  M. J. D. Powell,et al.  A method for nonlinear constraints in minimization problems , 1969 .

[34]  Apostol T. Vassilev,et al.  Analysis of the Inexact Uzawa Algorithm for Saddle Point Problems , 1997 .

[35]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[36]  I. Babuska.  The finite element method with Lagrangian multipliers , 1973 .

[37]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[38]  G. M. Korpelevich.  The extragradient method for finding saddle points and other problems , 1976 .

[39]  G. Golub,et al.  Inexact and preconditioned Uzawa algorithms for saddle point problems , 1994 .

[40]  J. Navarro-Pedreño.  Numerical Methods for Least Squares Problems , 1996 .

[41]  Tony F. Chan,et al.  A General Framework for a Class of First Order Primal-Dual Algorithms for Convex Optimization in Imaging Science , 2010, SIAM J. Imaging Sci..

[42]  Constantin Bacuta,et al.  A Unified Approach for Uzawa Algorithms , 2006, SIAM J. Numer. Anal..

[43]  Marc Teboulle,et al.  Fast Gradient-Based Algorithms for Constrained Total Variation Image Denoising and Deblurring Problems , 2009, IEEE Transactions on Image Processing.

[44]  Bingsheng He,et al.  Convergence Analysis of Primal-Dual Algorithms for a Saddle-Point Problem: From Contraction Perspective , 2012, SIAM J. Imaging Sci..

[45]  Raymond H. Chan,et al.  Affine Scaling Methods for Image Deblurring Problems , 2010 .

[46]  Y. Nesterov.  A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .