A first-order inexact primal-dual algorithm for a class of convex-concave saddle point problems

In this paper, we study a first-order inexact primal-dual algorithm (I-PDA) for solving a class of convex-concave saddle point problems. The I-PDA, which involves a relative error criterion and generalizes the classical PDA, has the advantage of solving one subproblem inexactly when it does not have a closed-form solution. We show that the whole sequence generated by I-PDA converges to a saddle point solution with $\mathcal {O}(1/N)$ ergodic convergence rate, where N is the iteration number. In addition, under a mild calmness condition, we establish the global Q-linear convergence rate of the distance between the iterates generated by I-PDA and the solution set, and the R-linear convergence speed of the nonergodic iterates. Furthermore, we demonstrate that many problems arising from practical applications satisfy this calmness condition. Finally, some numerical experiments are performed to show the superiority and linear convergence behaviors of I-PDA.

[1]  Bingsheng He,et al.  Convergence Analysis of Primal-Dual Algorithms for a Saddle-Point Problem: From Contraction Perspective , 2012, SIAM J. Imaging Sci..

[2]  Damek Davis,et al.  A Three-Operator Splitting Scheme and its Optimization Applications , 2015, 1504.01032.

[3]  Gauthier Gidel,et al.  Adaptive Three Operator Splitting , 2018, ICML.

[4]  Bingsheng He,et al.  An Algorithmic Framework of Generalized Primal–Dual Hybrid Gradient Methods for Saddle Point Problems , 2017, Journal of Mathematical Imaging and Vision.

[5]  Yonina C. Eldar,et al.  Smoothing and Decomposition for Analysis Sparse Recovery , 2013, IEEE Transactions on Signal Processing.

[6]  F ChanTony,et al.  A General Framework for a Class of First Order Primal-Dual Algorithms for Convex Optimization in Imaging Science , 2010 .

[7]  Jiaxin Xie,et al.  On inexact ADMMs with relative error criteria , 2018, Comput. Optim. Appl..

[8]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[9]  Xiaoming Yuan,et al.  A customized Douglas–Rachford splitting algorithm for separable convex minimization with linear constraints , 2013, Numerische Mathematik.

[10]  Michael Möller,et al.  The Primal-Dual Hybrid Gradient Method for Semiconvex Splittings , 2014, SIAM J. Imaging Sci..

[11]  Antonin Chambolle,et al.  Stochastic Primal-Dual Hybrid Gradient Algorithm with Arbitrary Sampling and Imaging Applications , 2017, SIAM J. Optim..

[12]  R. Rockafellar,et al.  Implicit Functions and Solution Mappings , 2009 .

[13]  Antonin Chambolle,et al.  On the ergodic convergence rates of a first-order primal–dual algorithm , 2016, Math. Program..

[14]  Julian Rasch,et al.  Inexact first-order primal–dual algorithms , 2018, Computational Optimization and Applications.

[15]  Ming Yan,et al.  A new primal-dual algorithm for minimizing the sum of three functions with a linear operator , 2016, 1611.09805.

[16]  Defeng Sun,et al.  Linear Rate Convergence of the Alternating Direction Method of Multipliers for Convex Composite Programming , 2017, Math. Oper. Res..

[17]  Michael Elad,et al.  The Cosparse Analysis Model and Algorithms , 2011, ArXiv.

[18]  Raymond H. Chan,et al.  Journal of Computational and Applied Mathematics a Reduced Newton Method for Constrained Linear Least-squares Problems , 2022 .

[19]  Laurent Condat,et al.  A Primal–Dual Splitting Method for Convex Optimization Involving Lipschitzian, Proximable and Linear Composite Terms , 2012, Journal of Optimization Theory and Applications.

[20]  Mingqiang Zhu,et al.  An Efficient Primal-Dual Hybrid Gradient Algorithm For Total Variation Image Restoration , 2008 .

[21]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[22]  Wang Yao,et al.  Approximate ADMM algorithms derived from Lagrangian splitting , 2017, Comput. Optim. Appl..

[23]  Deren Han,et al.  An improved first-order primal-dual algorithm with a new correction step , 2012, Journal of Global Optimization.

[24]  Thomas Pock,et al.  A First-Order Primal-Dual Algorithm with Linesearch , 2016, SIAM J. Optim..

[25]  S. M. Robinson An Implicit-Function Theorem for Generalized Variational Inequalities. , 1976 .

[26]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[27]  Lizhi Cheng,et al.  Precompact convergence of the nonconvex Primal-Dual Hybrid Gradient algorithm , 2018, J. Comput. Appl. Math..

[28]  Xiaoqun Zhang,et al.  A primal–dual fixed point algorithm for convex separable minimization with applications to image restoration , 2013 .

[29]  S. M. Robinson Some continuity properties of polyhedral multifunctions , 1981 .

[30]  Xiaoqun Zhang,et al.  A primal-dual fixed point algorithm for minimization of the sum of three convex separable functions , 2015, 1512.09235.

[31]  Fan Jiang,et al.  Approximate first-order primal-dual algorithms for saddle point problems , 2020, Math. Comput..

[32]  Wang Yao,et al.  Relative-error approximate versions of Douglas–Rachford splitting and special cases of the ADMM , 2018, Math. Program..

[33]  Tony F. Chan,et al.  A General Framework for a Class of First Order Primal-Dual Algorithms for Convex Optimization in Imaging Science , 2010, SIAM J. Imaging Sci..

[34]  B. V. Dean,et al.  Studies in Linear and Non-Linear Programming. , 1959 .

[35]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[36]  Yoram Singer,et al.  Efficient projections onto the l1-ball for learning in high dimensions , 2008, ICML '08.