A General Framework for a Class of First Order Primal-Dual Algorithms for Convex Optimization in Imaging Science

We generalize the primal-dual hybrid gradient (PDHG) algorithm proposed by Zhu and Chan in [An Efficient Primal-Dual Hybrid Gradient Algorithm for Total Variation Image Restoration, CAM Report 08-34, UCLA, Los Angeles, CA, 2008] to a broader class of convex optimization problems. In addition, we survey several closely related methods and explain the connections to PDHG. We point out convergence results for a modified version of PDHG that has a similarly good empirical convergence rate for total variation (TV) minimization problems. We also prove a convergence result for PDHG applied to TV denoising with some restrictions on the PDHG step size parameters. We show how to interpret this special case as a projected averaged gradient method applied to the dual functional. We discuss the range of parameters for which these methods can be shown to converge. We also present some numerical comparisons of these algorithms applied to TV denoising, TV deblurring, and constrained $l_1$ minimization problems.

[1]  Paul Tseng,et al.  Alternating Projection-Proximal Methods for Convex Programming and Variational Inequalities , 1997, SIAM J. Optim..

[2]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[3]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[4]  Michael Hintermüller,et al.  An Infeasible Primal-Dual Algorithm for Total Bounded Variation-Based Inf-Convolution-Type Image Restoration , 2006, SIAM J. Sci. Comput..

[5]  M. Nikolova.  An Algorithm for Total Variation Minimization and Applications , 2004 .

[6]  R. Glowinski,et al.  Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics , 1987 .

[7]  Dimitri P. Bertsekas,et al.  Constrained Optimization and Lagrange Multiplier Methods , 1982 .

[8]  Yiqiu Dong,et al.  An Efficient Primal-Dual Method for L1TV Image Restoration , 2009, SIAM J. Imaging Sci..

[9]  J. Moreau.  Proximité et dualité dans un espace hilbertien , 1965 .

[10]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[11]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[12]  Gene H. Golub,et al.  A Nonlinear Primal-Dual Method for Total Variation-Based Image Restoration , 1999, SIAM J. Sci. Comput..

[13]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[14]  Marc Teboulle,et al.  A proximal-based decomposition method for convex minimization problems , 1994, Math. Program..

[15]  Wotao Yin,et al.  Analysis and Generalizations of the Linearized Bregman Method , 2010, SIAM J. Imaging Sci..

[16]  丸山 徹.  Convex Analysisの二,三の進展について , 1977 .

[17]  Ernie Esser,et al.  Applications of Lagrangian-Based Alternating Direction Methods and Connections to Split Bregman , 2009 .

[18]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[19]  Jonathan Eckstein.  Splitting methods for monotone operators with applications to parallel optimization , 1989 .

[20]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[21]  P. Tseng.  Applications of splitting algorithm to decomposition in convex programming and variational inequalities , 1991 .

[22]  Karl Kunisch,et al.  Total Bounded Variation Regularization as a Bilaterally Constrained Optimization Problem , 2004, SIAM J. Appl. Math..

[23]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[24]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[25]  Emmanuel J. Candès,et al.  NESTA: A Fast and Accurate First-Order Method for Sparse Recovery , 2009, SIAM J. Imaging Sci..

[26]  R. Tyrrell Rockafellar,et al.  Augmented Lagrangians and Applications of the Proximal Point Algorithm in Convex Programming , 1976, Math. Oper. Res..

[27]  Wotao Yin,et al.  Bregman Iterative Algorithms for (cid:2) 1 -Minimization with Applications to Compressed Sensing ∗ , 2008 .

[28]  Daniel Cremers,et al.  An algorithm for minimizing the Mumford-Shah functional , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[29]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[30]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[31]  Marc Teboulle,et al.  Fast Gradient-Based Algorithms for Constrained Total Variation Image Denoising and Deblurring Problems , 2009, IEEE Transactions on Image Processing.

[32]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[33]  Yurii Nesterov,et al.  Dual extrapolation and its applications to solving variational inequalities and related problems , 2003, Math. Program..

[34]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[35]  A. Chambolle,et al.  An introduction to Total Variation for Image Analysis , 2009 .

[36]  Stanley Osher,et al.  A Unified Primal-Dual Algorithm Framework Based on Bregman Iteration , 2010, J. Sci. Comput..

[37]  Xue-Cheng Tai,et al.  Augmented Lagrangian Method, Dual Methods, and Split Bregman Iteration for ROF, Vectorial TV, and High Order Models , 2010, SIAM J. Imaging Sci..

[38]  Simon Setzer,et al.  Split Bregman Algorithm, Douglas-Rachford Splitting and Frame Shrinkage , 2009, SSVM.

[39]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[40]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[41]  I. Ekeland,et al.  Convex analysis and variational problems , 1976 .

[42]  Gregory B. Passty.  Ergodic convergence to a zero of the sum of monotone operators in Hilbert space , 1979 .

[43]  ZhangXiaoqun,et al.  A Unified Primal-Dual Algorithm Framework Based on Bregman Iteration , 2011 .

[44]  Xavier Bresson,et al.  Bregmanized Nonlocal Regularization for Deconvolution and Sparse Reconstruction , 2010, SIAM J. Imaging Sci..

[45]  Gilles Aubert,et al.  Efficient Schemes for Total Variation Minimization Under Constraints in Image Processing , 2009, SIAM J. Sci. Comput..

[46]  Mingqiang Zhu,et al.  An Efficient Primal-Dual Hybrid Gradient Algorithm For Total Variation Image Restoration , 2008 .

[47]  H. H. Rachford,et al.  On the numerical solution of heat conduction problems in two and three space variables , 1956 .

[48]  Junfeng Yang,et al.  A New Alternating Minimization Algorithm for Total Variation Image Reconstruction , 2008, SIAM J. Imaging Sci..

[49]  Stephen J. Wright,et al.  Duality-based algorithms for total-variation-regularized image restoration , 2010, Comput. Optim. Appl..

[50]  Justin Romberg,et al.  Practical Signal Recovery from Random Projections , 2005 .

[51]  Emmanuel J. Candès,et al.  Signal recovery from random projections , 2005, IS&T/SPIE Electronic Imaging.

[52]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[53]  Tom Goldstein,et al.  The Split Bregman Method for L1-Regularized Problems , 2009, SIAM J. Imaging Sci..

[54]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[55]  Wotao Yin,et al.  Second-order Cone Programming Methods for Total Variation-Based Image Restoration , 2005, SIAM J. Sci. Comput..

[56]  F ChanTony,et al.  A General Framework for a Class of First Order Primal-Dual Algorithms for Convex Optimization in Imaging Science , 2010 .

[57]  L. Popov.  A modification of the Arrow-Hurwicz method for search of saddle points , 1980 .

[58]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[59]  Michael Patriksson,et al.  On the convergence of conditional epsilon-subgradient methods for convex programs and convex-concave saddle-point problems , 2003, Eur. J. Oper. Res..

[60]  Abderrahim Elmoataz,et al.  Nonlocal Discrete Regularization on Weighted Graphs: A Framework for Image and Manifold Processing , 2008, IEEE Transactions on Image Processing.