Linear Convergence and Metric Selection for Douglas-Rachford Splitting and ADMM

Recently, several convergence rate results for Douglas-Rachford splitting and the alternating direction method of multipliers (ADMM) have been presented in the literature. In this paper, we show global linear convergence rate bounds for Douglas-Rachford splitting and ADMM under strong convexity and smoothness assumptions. We further show that the rate bounds are tight for the class of problems under consideration for all feasible algorithm parameters. For problems that satisfy the assumptions, we show how to select step-size and metric for the algorithm that optimize the derived convergence rate bounds. For problems with a similar structure that do not satisfy the assumptions, we present heuristic step-size and metric selection methods.

[1]  Stephen P. Boyd,et al.  Line search for averaged operator iteration , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[2]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..

[3]  Stephen P. Boyd,et al.  Metric selection in fast dual forward-backward splitting , 2015, Autom..

[4]  Pontus Giselsson,et al.  Tight global linear convergence rate bounds for Douglas–Rachford splitting , 2015, Journal of Fixed Point Theory and Applications.

[5]  Pontus Giselsson,et al.  Tight linear convergence rate bounds for Douglas-Rachford splitting and ADMM , 2015, 2015 54th IEEE Conference on Decision and Control (CDC).

[6]  Kristian Bredies,et al.  Preconditioned Douglas-Rachford Splitting Methods for Convex-concave Saddle-point Problems , 2015, SIAM J. Numer. Anal..

[7]  Michael I. Jordan,et al.  A General Analysis of the Convergence of ADMM , 2015, ICML.

[8]  Stephen P. Boyd,et al.  Diagonal scaling in Douglas-Rachford splitting and ADMM , 2014, 53rd IEEE Conference on Decision and Control.

[9]  A. Raghunathan,et al.  ADMM for Convex Quadratic Programs: Linear Convergence and Infeasibility Detection , 2014, 1411.7288.

[10]  Alberto Bemporad,et al.  Douglas-rachford splitting: Complexity estimates and accelerated variants , 2014, 53rd IEEE Conference on Decision and Control.

[11]  Wotao Yin,et al.  Faster Convergence Rates of Relaxed Peaceman-Rachford and ADMM Under Regularity Assumptions , 2014, Math. Oper. Res..

[12]  Damek Davis,et al.  Convergence Rate Analysis of Several Splitting Schemes , 2014, 1406.4834.

[13]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[14]  Hung M. Phan,et al.  Linear convergence of the Douglas–Rachford method for two closed sets , 2014, 1401.6509.

[15]  Pascal Bianchi,et al.  Explicit Convergence Rate of a Distributed Alternating Direction Method of Multipliers , 2013, IEEE Transactions on Automatic Control.

[16]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[17]  Daniel Boley,et al.  Local Linear Convergence of the Alternating Direction Method of Multipliers on Quadratic or Linear Programs , 2013, SIAM J. Optim..

[18]  Heinz H. Bauschke,et al.  The rate of linear convergence of the Douglas-Rachford algorithm for subspaces is the cosine of the Friedrichs angle , 2013, J. Approx. Theory.

[19]  Qing Ling,et al.  On the Linear Convergence of the ADMM in Decentralized Consensus Optimization , 2013, IEEE Transactions on Signal Processing.

[20]  D. Russell Luke,et al.  Alternating Projections and Douglas-Rachford for Sparse Affine Feasibility , 2013, IEEE Transactions on Signal Processing.

[21]  Euhanna Ghadimi,et al.  Optimal Parameter Selection for the Alternating Direction Method of Multipliers (ADMM): Quadratic Problems , 2013, IEEE Transactions on Automatic Control.

[22]  Stephen P. Boyd,et al.  A Splitting Method for Optimal Control , 2013, IEEE Transactions on Control Systems Technology.

[23]  Laurent Demanet,et al.  Eventual linear convergence of the Douglas-Rachford iteration for basis pursuit , 2013, Math. Comput..

[24]  D. Russell Luke,et al.  Nonconvex Notions of Regularity and Convergence of Fundamental Algorithms for Feasibility Problems , 2012, SIAM J. Optim..

[25]  Zhi-Quan Luo,et al.  On the linear convergence of the alternating direction method of multipliers , 2012, Mathematical Programming.

[26]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[27]  Antonin Chambolle,et al.  Diagonal preconditioning for first order primal-dual algorithms in convex optimization , 2011, 2011 International Conference on Computer Vision.

[28]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[29]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[30]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[31]  Patrick L. Combettes,et al.  Proximal Splitting Methods in Signal Processing , 2009, Fixed-Point Algorithms for Inverse Problems in Science and Engineering.

[32]  D. Donoho,et al.  Sparse MRI: The application of compressed sensing for rapid MR imaging , 2007, Magnetic resonance in medicine.

[33]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[34]  Jun Zou,et al.  Nonlinear Inexact Uzawa Algorithms for Linear and Nonlinear Saddle-point Problems , 2006, SIAM J. Optim..

[35]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[36]  E. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[37]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[38]  M. Benzi.  Preconditioning techniques for large linear systems: a survey , 2002 .

[39]  Apostol T. Vassilev,et al.  Analysis of the Inexact Uzawa Algorithm for Saddle Point Problems , 1997 .

[40]  E. Mosca,et al.  Nonlinear control of constrained linear systems via predictive reference management , 1997, IEEE Trans. Autom. Control..

[41]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[42]  Michael Athans,et al.  Design of feedback control systems for stable plants with saturating actuators , 1988, Proceedings of the 27th IEEE Conference on Decision and Control.

[43]  P. Lions,et al.  Splitting Algorithms for the Sum of Two Nonlinear Operators , 1979 .

[44]  B. V. Dean,et al.  Studies in Linear and Non-Linear Programming. , 1959 .

[45]  H. H. Rachford,et al.  On the numerical solution of heat conduction problems in two and three space variables , 1956 .

[46]  H. H. Rachford,et al.  The Numerical Solution of Parabolic and Elliptic Differential Equations , 1955 .

[47]  Bastian Goldlücke,et al.  Variational Analysis , 2014, Computer Vision, A Reference Guide.

[48]  Khadir Mohamed,et al.  Model Predictive Control: Theory and Design , 2014 .

[49]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[50]  Patrick L. Combettes,et al.  Signal Recovery by Proximal Forward-Backward Splitting , 2005, Multiscale Model. Simul..

[51]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[52]  Jonathan Eckstein.  Splitting methods for monotone operators with applications to parallel optimization , 1989 .

[53]  D. Gabay.  Applications of the method of multipliers to variational inequalities , 1983 .

[54]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[55]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .