Comparison of Proximal First-Order Primal and Primal-Dual algorithms via Performance Estimation

Selecting the fastest algorithm for a specific signal/image processing task is a challenging question. We propose an approach based on the Performance Estimation Problem framework that numerically and automatically computes the worst-case performance of a given optimization method on a class of functions. We first propose a computer-assisted analysis and comparison of several first-order primal optimization methods, namely, the gradient method, the forward-backward, Peaceman-Rachford, and Douglas-Rachford splittings. We tighten the existing convergence results of these algorithms and extend them to new classes of functions. Our analysis is then extended and evaluated in the context of the primal-dual Chambolle-Pock and Condat-V\~u methods.

[1]  Franccois Glineur,et al.  Interpolation Conditions for Linear Operators and Applications to Performance Estimation Problems , 2023, 2302.08781.

[2]  N. Pustelnik,et al.  Theoretical and numerical comparison of first order algorithms for cocoercive equations and smooth convex optimization , 2021, Signal Process..

[3]  Laurent Condat,et al.  Proximal Splitting Algorithms for Convex Optimization: A Tour of Recent Advances, with New Twists , 2019, SIAM Rev..

[4]  Adrien B. Taylor,et al.  Operator Splitting Performance Estimation: Tight Contraction Factors and Optimal Parameter Selection , 2018, SIAM J. Optim..

[5]  Amir Beck,et al.  First-Order Methods in Optimization , 2017 .

[6]  Antonin Chambolle,et al.  An introduction to continuous optimization for imaging , 2016, Acta Numerica.

[7]  J. Pesquet,et al.  Wavelet-Based Image Deconvolution and Reconstruction , 2016 .

[8]  Adrien B. Taylor,et al.  Exact Worst-Case Performance of First-Order Methods for Composite Convex Optimization , 2015, SIAM J. Optim..

[9]  Adrien B. Taylor,et al.  Smooth strongly convex interpolation and exact worst-case performance of first-order methods , 2015, Mathematical Programming.

[10]  Laurent Condat,et al.  A Primal–Dual Splitting Method for Convex Optimization Involving Lipschitzian, Proximable and Linear Composite Terms , 2012, Journal of Optimization Theory and Applications.

[11]  Marc Teboulle,et al.  Performance of first-order methods for smooth convex minimization: a novel approach , 2012, Mathematical Programming.

[12]  Bang Công Vu,et al.  A splitting algorithm for dual monotone inclusions involving cocoercive operators , 2011, Advances in Computational Mathematics.

[13]  Julien Mairal,et al.  Optimization with Sparsity-Inducing Penalties , 2011, Found. Trends Mach. Learn..

[14]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[15]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[16]  R. Tyrrell Rockafellar,et al.  Convergence Rates in Forward-Backward Splitting , 1997, SIAM J. Optim..

[17]  L. Briceño-Arias,et al.  Convergence Rate Comparison of Proximal Algorithms for Non-Smooth Convex Optimization With an Application to Texture Segmentation , 2022, IEEE Signal Processing Letters.

[18]  A. Beck,et al.  Gradient-based algorithms with applications to signal-recovery problems , 2010, Convex Optimization in Signal Processing and Communications.

[19]  Arkadi Nemirovski,et al.  Prox-Method with Rate of Convergence O(1/t) for Variational Inequalities with Lipschitz Continuous Monotone Operators and Smooth Convex-Concave Saddle Point Problems , 2004, SIAM J. Optim..