Convergence Rate Analysis of Primal-Dual Splitting Schemes

Primal-dual splitting schemes are a class of powerful algorithms that solve complicated monotone inclusions and convex optimization problems that are built from many simpler pieces. They decompose problems that are built from sums, linear compositions, and infimal convolutions of simple functions so that each simple term is processed individually via proximal mappings, gradient mappings, and multiplications by the linear maps. This leads to easily implementable and highly parallelizable or distributed algorithms, which often obtain nearly state-of-the-art performance. In this paper, we analyze a monotone inclusion problem that captures a large class of primal-dual splittings as a special case. We introduce a unifying scheme and use some abstract analysis of the algorithm to prove convergence rates of the proximal point algorithm, forward-backward splitting, Peaceman--Rachford splitting, and forward-backward-forward splitting applied to the model problem. Our ergodic convergence rates are deduced under var...

[1]  P. L. Combettes,et al.  Variable metric quasi-Fejér monotonicity , 2012, 1206.5705.

[2]  Daniel Cremers,et al.  An algorithm for minimizing the Mumford-Shah functional , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[3]  Tosio Kato.  Perturbation theory for linear operators , 1966 .

[4]  Marc Teboulle,et al.  Rate of Convergence Analysis of Decomposition Methods Based on the Proximal Method of Multipliers for Convex Minimization , 2014, SIAM J. Optim..

[5]  Xiaoqun Zhang,et al.  A primal–dual fixed point algorithm for convex separable minimization with applications to image restoration , 2013 .

[6]  Patrick L. Combettes,et al.  A Monotone+Skew Splitting Model for Composite Monotone Inclusions in Duality , 2010, SIAM J. Optim..

[7]  Radu Ioan Bot,et al.  Convergence Analysis for a Primal-Dual Monotone + Skew Splitting Algorithm with Applications to Total Variation Minimization , 2012, Journal of Mathematical Imaging and Vision.

[8]  P. L. Combettes,et al.  Variable metric forward–backward splitting with applications to monotone inclusions in duality , 2012, 1206.6791.

[9]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[10]  Radu Ioan Bot,et al.  A Douglas-Rachford Type Primal-Dual Method for Solving Inclusions with Mixtures of Composite and Parallel-Sum Type Monotone Operators , 2012, SIAM J. Optim..

[11]  J. Baillon,et al.  Quelques propriétés des opérateurs angle-bornés etn-cycliquement monotones , 1977 .

[12]  Patrick L. Combettes,et al.  Systems of Structured Monotone Inclusions: Duality, Algorithms, and Applications , 2012, SIAM J. Optim..

[13]  R. Boţ,et al.  Solving monotone inclusions involving parallel sums of linearly composed maximally monotone operators , 2013, 1306.3191.

[14]  Patrick L. Combettes,et al.  An Algorithm for Splitting Parallel Sums of Linearly Composed Monotone Operators, with Applications to Signal Recovery , 2013, 1305.5828.

[15]  Gregory B. Passty.  Ergodic convergence to a zero of the sum of monotone operators in Hilbert space , 1979 .

[16]  Ronald E. Bruck.  On the weak convergence of an ergodic iteration for the solution of variational inequalities for monotone operators in Hilbert space , 1977 .

[17]  Dimitri P. Bertsekas,et al.  Incremental Gradient, Subgradient, and Proximal Methods for Convex Optimization: A Survey , 2015, ArXiv.

[18]  B. Vu.  A Variable Metric Extension of the Forward–Backward–Forward Algorithm for Monotone Operators , 2012, 1210.2986.

[19]  Yurii Nesterov,et al.  Smooth minimization of non-smooth functions , 2005, Math. Program..

[20]  Damek Davis,et al.  Convergence Rate Analysis of Several Splitting Schemes , 2014, 1406.4834.

[21]  Pablo A. Lotito,et al.  A Class of Inexact Variable Metric Proximal Point Algorithms , 2008, SIAM J. Optim..

[22]  P. L. Combettes,et al.  Primal-Dual Splitting Algorithm for Solving Inclusions with Mixtures of Composite, Lipschitzian, and Parallel-Sum Type Monotone Operators , 2011, Set-Valued and Variational Analysis.

[23]  Radu Ioan Bot,et al.  On the convergence rate improvement of a primal-dual splitting algorithm for solving monotone inclusion problems , 2013, Mathematical Programming.

[24]  Laurent Condat,et al.  A Primal–Dual Splitting Method for Convex Optimization Involving Lipschitzian, Proximable and Linear Composite Terms , 2012, Journal of Optimization Theory and Applications.

[25]  P. Lions,et al.  Image recovery via total variation minimization and related problems , 1997 .

[26]  Paul Tseng,et al.  A Modified Forward-backward Splitting Method for Maximal Monotone Mappings 1 , 1998 .

[27]  D. Varberg.  Convex Functions , 1973 .

[28]  Patrick L. Combettes,et al.  A forward-backward view of some primal-dual optimization methods in image recovery , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[29]  Xiaoming Yuan,et al.  A Generalized Proximal Point Algorithm and Its Convergence Rate , 2014, SIAM J. Optim..

[30]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[31]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[32]  Lieven Vandenberghe,et al.  Primal-Dual Decomposition by Operator Splitting and Applications to Image Deblurring , 2014, SIAM J. Imaging Sci..

[33]  R. Boţ,et al.  Solving Systems of Monotone Inclusions via Primal-dual Splitting Techniques , 2013 .

[34]  Antonin Chambolle,et al.  Diagonal preconditioning for first order primal-dual algorithms in convex optimization , 2011, 2011 International Conference on Computer Vision.

[35]  Tony F. Chan,et al.  A General Framework for a Class of First Order Primal-Dual Algorithms for Convex Optimization in Imaging Science , 2010, SIAM J. Imaging Sci..

[36]  J. Borwein,et al.  Convex Functions: Constructions, Characterizations and Counterexamples , 2010 .

[37]  J. Pesquet,et al.  A Class of Randomized Primal-Dual Algorithms for Distributed Optimization , 2014, 1406.6404.

[38]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[39]  Steven M. LaValle,et al.  Planning algorithms , 2006 .

[40]  Bingsheng He,et al.  Convergence Analysis of Primal-Dual Algorithms for a Saddle-Point Problem: From Contraction Perspective , 2012, SIAM J. Imaging Sci..

[41]  Julien Mairal,et al.  Proximal Methods for Hierarchical Sparse Coding , 2010, J. Mach. Learn. Res..

[42]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[43]  Bang Công Vu,et al.  A splitting algorithm for dual monotone inclusions involving cocoercive operators , 2011, Advances in Computational Mathematics.

[44]  I. Yamada,et al.  NON-STRICTLY CONVEX MINIMIZATION OVER THE FIXED POINT SET OF AN ASYMPTOTICALLY SHRINKING NONEXPANSIVE MAPPING , 2002 .

[45]  Ernö Robert Csetnek,et al.  On the convergence rate of a forward-backward type primal-dual splitting algorithm for convex optimization problems , 2015 .

[46]  Xiaoming Yuan,et al.  Adaptive Primal-Dual Hybrid Gradient Methods for Saddle-Point Problems , 2013, 1305.0546.