A Generalized Primal-Dual Algorithm with Improved Convergence Condition for Saddle Point Problems

We generalize the well-known primal-dual algorithm proposed by Chambolle and Pock for saddle point problems, and improve the condition for ensuring its convergence. The improved convergence-guaranteeing condition is effective for the generic setting, and it is shown to be optimal. It also allows us to discern larger step sizes for the resulting subproblems, and thus provides a simple and universal way to improve numerical performance of the original primal-dual algorithm. In addition, we present a structure-exploring heuristic to further relax the convergence-guaranteeing condition for some specific saddle point problems, which could yield much larger step sizes and hence significantly better performance. Effectiveness of this heuristic is numerically illustrated by the classic assignment problem.

[1]  Amir Beck,et al.  First-Order Methods in Optimization , 2017 .

[2]  Phillipp Kaestner,et al.  Linear And Nonlinear Programming , 2016 .

[3]  Antonin Chambolle,et al.  An introduction to continuous optimization for imaging , 2016, Acta Numerica.

[4]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[5]  Antonin Chambolle,et al.  On the ergodic convergence rates of a first-order primal–dual algorithm , 2016, Math. Program..

[6]  Xue-Cheng Tai,et al.  Efficient and Convergent Preconditioned ADMM for the Potts Models , 2021, SIAM J. Sci. Comput..

[7]  Stanley Osher,et al.  A Unified Primal-Dual Algorithm Framework Based on Bregman Iteration , 2010, J. Sci. Comput..

[8]  Bingsheng He,et al.  On non-ergodic convergence rate of Douglas–Rachford alternating direction method of multipliers , 2014, Numerische Mathematik.

[9]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[10]  Laurent Condat,et al.  A Primal–Dual Splitting Method for Convex Optimization Involving Lipschitzian, Proximable and Linear Composite Terms , 2012, Journal of Optimization Theory and Applications.

[11]  Valeria Ruggiero,et al.  On the Convergence of Primal–Dual Hybrid Gradient Algorithms for Total Variation Image Restoration , 2012, Journal of Mathematical Imaging and Vision.

[12]  Xiaoming Yuan,et al.  Optimal proximal augmented Lagrangian method and its application to full Jacobian splitting for multi-block separable convex minimization problems , 2020, IMA Journal of Numerical Analysis.

[13]  Bingsheng He,et al.  Convergence Analysis of Primal-Dual Algorithms for a Saddle-Point Problem: From Contraction Perspective , 2012, SIAM J. Imaging Sci..

[14]  Mingqiang Zhu,et al.  An Efficient Primal-Dual Hybrid Gradient Algorithm For Total Variation Image Restoration , 2008 .

[15]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[16]  Antonin Chambolle,et al.  Diagonal preconditioning for first order primal-dual algorithms in convex optimization , 2011, 2011 International Conference on Computer Vision.

[17]  B. V. Dean,et al.  Studies in Linear and Non-Linear Programming. , 1959 .

[18]  Xue-Cheng Tai,et al.  A Continuous Max-Flow Approach to Potts Model , 2010, ECCV.

[19]  Alexander Schrijver,et al.  Theory of linear and integer programming , 1986, Wiley-Interscience series in discrete mathematics and optimization.

[20]  Bingsheng He,et al.  On the Convergence of Primal-Dual Hybrid Gradient Algorithm , 2014, SIAM J. Imaging Sci..

[21]  Tony F. Chan,et al.  A General Framework for a Class of First Order Primal-Dual Algorithms for Convex Optimization in Imaging Science , 2010, SIAM J. Imaging Sci..

[22]  F. Facchinei,et al.  Finite-Dimensional Variational Inequalities and Complementarity Problems , 2003 .

[23]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[24]  Bingsheng He,et al.  An Algorithmic Framework of Generalized Primal–Dual Hybrid Gradient Methods for Saddle Point Problems , 2017, Journal of Mathematical Imaging and Vision.

[25]  Michael Elad,et al.  From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images , 2009, SIAM Rev..

[26]  A. Chambolle,et al.  An introduction to Total Variation for Image Analysis , 2009 .

[27]  OsherStanley,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[28]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[29]  Thomas Pock,et al.  A First-Order Primal-Dual Algorithm with Linesearch , 2016, SIAM J. Optim..