Distributed Proximal Splitting Algorithms with Rates and Acceleration

We analyze several generic proximal splitting algorithms well suited for large-scale convex nonsmooth optimization. We derive sublinear and linear convergence results with new rates on the function value suboptimality or distance to the solution, as well as new accelerated versions, using varying stepsizes. In addition, we propose distributed variants of these algorithms, which can be accelerated as well. While most existing results are ergodic, our nonergodic results significantly broaden our understanding of primal–dual optimization algorithms.

[1]  Jean-Yves Audibert.  Optimization for Machine Learning , 1995 .

[2]  Stephen P. Boyd,et al.  Trend Filtering ∗ , 2009 .

[3]  D. Cremers.  Convex Relaxation Techniques for Segmentation , Stereo and Multiview Reconstruction , 2010 .

[4]  Convex Optimization in Signal Processing and Communications , 2010 .

[5]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[6]  Karl Kunisch,et al.  Total Generalized Variation , 2010, SIAM J. Imaging Sci..

[7]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[8]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[9]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[10]  P. L. Combettes,et al.  Primal-Dual Splitting Algorithm for Solving Inclusions with Mixtures of Composite, Lipschitzian, and Parallel-Sum Type Monotone Operators , 2011, Set-Valued and Variational Analysis.

[11]  I. Loris,et al.  On a generalization of the iterative soft-thresholding algorithm for the case of non-separable penalty , 2011, 1104.1087.

[12]  F. Bach,et al.  Optimization with Sparsity-Inducing Penalties (Foundations and Trends(R) in Machine Learning) , 2011 .

[13]  Julien Mairal,et al.  Optimization with Sparsity-Inducing Penalties , 2011, Found. Trends Mach. Learn..

[14]  Laurent Condat,et al.  A Primal–Dual Splitting Method for Convex Optimization Involving Lipschitzian, Proximable and Linear Composite Terms , 2012, Journal of Optimization Theory and Applications.

[15]  Bang Công Vu,et al.  A splitting algorithm for dual monotone inclusions involving cocoercive operators , 2011, Advances in Computational Mathematics.

[16]  Xiaoqun Zhang,et al.  A primal–dual fixed point algorithm for convex separable minimization with applications to image restoration , 2013 .

[17]  Volkan Cevher,et al.  Convex Optimization for Big Data: Scalable, randomized, and parallel algorithms for big data analytics , 2014, IEEE Signal Processing Magazine.

[18]  Laurent Condat,et al.  A Generic Proximal Algorithm for Convex Optimization—Application to Total Variation Minimization , 2014, IEEE Signal Processing Letters.

[19]  Peter Richtárik,et al.  Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function , 2011, Mathematical Programming.

[20]  Patrick L. Combettes,et al.  A forward-backward view of some primal-dual optimization methods in image recovery , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[21]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[22]  Ernö Robert Csetnek,et al.  Recent Developments on Primal–Dual Splitting Methods with Applications to Convex Minimization , 2014 .

[23]  Marc Teboulle,et al.  A simple algorithm for a class of nonsmooth convex-concave saddle-point problems , 2015, Oper. Res. Lett..

[24]  Sébastien Bubeck,et al.  Convex Optimization: Algorithms and Complexity , 2014, Found. Trends Mach. Learn..

[25]  James G. Scott,et al.  Proximal Algorithms in Statistics and Machine Learning , 2015, ArXiv.

[26]  Damek Davis,et al.  A Three-Operator Splitting Scheme and its Optimization Applications , 2015, 1504.01032.

[27]  Nikos Komodakis,et al.  Playing with Duality: An overview of recent primal?dual approaches for solving large-scale optimization problems , 2014, IEEE Signal Process. Mag..

[28]  Qing Ling,et al.  EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization , 2014, 1404.6264.

[29]  Antonin Chambolle,et al.  An introduction to continuous optimization for imaging , 2016, Acta Numerica.

[30]  Peter Richtárik,et al.  Federated Learning: Strategies for Improving Communication Efficiency , 2016, ArXiv.

[31]  Antonin Chambolle,et al.  On the ergodic convergence rates of a first-order primal–dual algorithm , 2016, Math. Program..

[32]  Colin N. Jones,et al.  Operator Splitting Methods in Control , 2016, Found. Trends Syst. Control..

[33]  Alexander J. Smola,et al.  Trend Filtering on Graphs , 2014, J. Mach. Learn. Res..

[34]  Michael Möller,et al.  Collaborative Total Variation: A General Framework for Vectorial TV Models , 2015, SIAM J. Imaging Sci..

[35]  Laurent Condat,et al.  Discrete Total Variation: New Definition and Minimization , 2017, SIAM J. Imaging Sci..

[36]  Amir Beck,et al.  First-Order Methods in Optimization , 2017 .

[37]  Laurent Massoulié,et al.  Optimal Algorithms for Smooth and Strongly Convex Distributed Optimization in Networks , 2017, ICML.

[38]  Wotao Yin,et al.  Splitting Methods in Communication, Imaging, Science, and Engineering , 2017 .

[39]  Laurent Condat,et al.  A Convex Approach to K-Means Clustering and Image Segmentation , 2017, EMMCVPR.

[40]  Ming Yan,et al.  A new primal-dual algorithm for minimizing the sum of three functions with a linear operator , 2016, 1611.09805.

[41]  Laurent Condat,et al.  A Convex Lifting Approach to Image Phase Unwrapping , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[42]  Panagiotis Patrinos,et al.  A New Randomized Block-Coordinate Primal-Dual Proximal Algorithm for Distributed Optimization , 2017, IEEE Transactions on Automatic Control.

[43]  Proximal Splitting Algorithms for Convex Optimization: A Tour of Recent Advances, with New Twists , 2019, 1912.00137.

[44]  Laurent Condat,et al.  Proximal splitting algorithms: Relax them all! , 2019 .

[45]  Ernest K. Ryu,et al.  Proximal-Proximal-Gradient Method , 2017, Journal of Computational Mathematics.

[46]  Laurent Condat,et al.  Proximal Splitting Algorithms: A Tour of Recent Advances, with New Twists. , 2020 .

[47]  Peter Richtárik,et al.  Dualize, Split, Randomize: Fast Nonsmooth Optimization Algorithms , 2020, arXiv.org.

[48]  Peter Richtárik,et al.  From Local SGD to Local Fixed Point Methods for Federated Learning , 2020, ICML.

[49]  Lieven Vandenberghe,et al.  On the equivalence of the primal-dual hybrid gradient method and Douglas–Rachford splitting , 2018, Math. Program..

[50]  Peter Richtárik,et al.  A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descent , 2019, AISTATS.

[51]  Ali H. Sayed,et al.  Decentralized Proximal Gradient Algorithms With Linear Convergence Rates , 2019, IEEE Transactions on Automatic Control.

[52]  Peter Richtárik,et al.  An Optimal Algorithm for Strongly Convex Minimization under Affine Constraints , 2021, AISTATS.