Linear Convergence Rate of a Class of Distributed Augmented Lagrangian Algorithms

We study distributed optimization where nodes cooperatively minimize the sum of their individual, locally known, convex costs fi(x)'s; x ϵ ℝd is global. Distributed augmented Lagrangian (AL) methods have good empirical performance on several signal processing and learning applications, but there is limited understanding of their convergence rates and how it depends on the underlying network. This paper establishes globally linear (geometric) convergence rates of a class of deterministic and randomized distributed AL methods, when the fi's are twice continuously differentiable and have a bounded Hessian. We give explicit dependence of the convergence rates on the underlying network parameters. Simulations illustrate our analytical findings.

[1]  B. V. Dean,et al.  Studies in Linear and Non-Linear Programming. , 1959 .

[2]  M. Powell.  A method for nonlinear constraints in minimization problems , 1969 .

[3]  M. Hestenes.  Multiplier and gradient methods , 1969 .

[4]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[5]  R. Glowinski,et al.  Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires , 1975 .

[6]  D. Bertsekas,et al.  Combined Primal–Dual and Penalty Methods for Convex Programming , 1976 .

[7]  B. Mercier,et al.  A dual algorithm for the solution of nonlinear variational problems via finite element approximation , 1976 .

[8]  R. Tyrrell Rockafellar,et al.  Augmented Lagrangians and Applications of the Proximal Point Algorithm in Convex Programming , 1976, Math. Oper. Res..

[9]  M. Tarazi.  Some convergence results for asynchronous algorithms , 1982 .

[10]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[11]  P. Tseng,et al.  On the linear convergence of descent methods for convex essentially smooth minimization , 1992 .

[12]  Dimitri P. Bertsekas,et al.  On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators , 1992, Math. Program..

[13]  A. Ruszczynski,et al.  Perturbation Methods for Saddle Point Computation , 1994 .

[14]  Fan Chung,et al.  Spectral Graph Theory , 1996 .

[15]  Xiaojun Chen.  Global and Superlinear Convergence of Inexact Uzawa Methods for Saddle Point Problems with Nondiffer , 1998 .

[16]  Apostol T. Vassilev,et al.  Analysis of the Inexact Uzawa Algorithm for Saddle Point Problems , 1997 .

[17]  E. Kaszkurewicz,et al.  Matrix diagonal stability in systems and computation , 1999 .

[18]  Claude Lemaréchal,et al.  Lagrangian Relaxation , 2000, Computational Combinatorial Optimization.

[19]  Robert Nowak,et al.  Distributed optimization in sensor networks , 2004, Third International Symposium on Information Processing in Sensor Networks, 2004. IPSN 2004.

[20]  Jun Zou,et al.  Nonlinear Inexact Uzawa Algorithms for Linear and Nonlinear Saddle-point Problems , 2006, SIAM J. Optim..

[21]  Alejandro Ribeiro,et al.  Consensus in Ad Hoc WSNs With Noisy Links—Part I: Distributed Estimation of Deterministic Signals , 2008, IEEE Transactions on Signal Processing.

[22]  Stergios I. Roumeliotis,et al.  Consensus in Ad Hoc WSNs With Noisy Links—Part II: Distributed Estimation and Smoothing of Random Signals , 2008, IEEE Transactions on Signal Processing.

[23]  Asuman E. Ozdaglar,et al.  Distributed Subgradient Methods for Multi-Agent Optimization , 2009, IEEE Transactions on Automatic Control.

[24]  Angelia Nedic,et al.  Subgradient Methods for Saddle-Point Problems , 2009, J. Optimization Theory and Applications.

[25]  Georgios B. Giannakis,et al.  Distributed Spectrum Sensing for Cognitive Radio Networks by Exploiting Sparsity , 2010, IEEE Transactions on Signal Processing.

[26]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[27]  Jing Wang,et al.  Control approach to distributed optimization , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[28]  Gonzalo Mateos,et al.  Distributed Sparse Linear Regression , 2010, IEEE Transactions on Signal Processing.

[29]  José M. F. Moura,et al.  Cooperative Convex Optimization in Networked Systems: Augmented Lagrangian Algorithms With Directed Gossip Communication , 2010, IEEE Transactions on Signal Processing.

[30]  Jing Wang,et al.  A control perspective for centralized and distributed convex optimization , 2011, IEEE Conference on Decision and Control and European Control Conference.

[31]  R. Murray,et al.  Decentralized Multi-Agent Optimization via Dual Decomposition , 2011 .

[32]  Euhanna Ghadimi,et al.  Accelerated gradient methods for networked optimization , 2011, Proceedings of the 2011 American Control Conference.

[33]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[34]  Emiliano Dall'Anese,et al.  Fast Consensus by the Alternating Direction Multipliers Method , 2011, IEEE Transactions on Signal Processing.

[35]  John S. Baras,et al.  Performance Evaluation of the Consensus-Based Distributed Subgradient Method Under Random Communication Topologies , 2011, IEEE Journal of Selected Topics in Signal Processing.

[36]  Soummya Kar,et al.  Distributed Parameter Estimation in Sensor Networks: Nonlinear Observation Models and Imperfect Communication , 2008, IEEE Transactions on Information Theory.

[37]  Asuman E. Ozdaglar,et al.  Distributed Alternating Direction Method of Multipliers , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[38]  Junfeng Lu.  Convergence Analysis of the Modified Nonlinear Inexact Uzawa Algorithm for Saddle Point Problem , 2012 .

[39]  Sonia Martínez,et al.  On Distributed Convex Optimization Under Inequality and Equality Constraints , 2010, IEEE Transactions on Automatic Control.

[40]  Martin J. Wainwright,et al.  Dual Averaging for Distributed Optimization: Convergence Analysis and Network Scaling , 2010, IEEE Transactions on Automatic Control.

[41]  João M. F. Xavier,et al.  Distributed Basis Pursuit , 2010, IEEE Transactions on Signal Processing.

[42]  Bahman Gharesifard,et al.  Continuous-time distributed convex optimization on weight-balanced digraphs , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[43]  Qing Ling,et al.  Linearly convergent decentralized consensus optimization with the alternating direction method of multipliers , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[44]  José M. F. Moura,et al.  Fast Distributed Gradient Methods , 2011, IEEE Transactions on Automatic Control.

[45]  Ion Necoara,et al.  Computational Complexity of Inexact Gradient Augmented Lagrangian Methods: Application to Constrained MPC , 2013, SIAM J. Control. Optim..

[46]  Bahman Gharesifard,et al.  Distributed Continuous-Time Convex Optimization on Weight-Balanced Digraphs , 2012, IEEE Transactions on Automatic Control.

[47]  Qing Ling,et al.  On the Linear Convergence of the ADMM in Decentralized Consensus Optimization , 2013, IEEE Transactions on Signal Processing.

[48]  M. Malek.  Vector Calculus , 2014 .

[49]  Renato D. C. Monteiro,et al.  Iteration-complexity of first-order augmented Lagrangian methods for convex programming , 2015, Mathematical Programming.

[50]  Wotao Yin,et al.  On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers , 2016, J. Sci. Comput..

[51]  Qing Ling,et al.  On the Convergence of Decentralized Gradient Descent , 2013, SIAM J. Optim..

[52]  Zhi-Quan Luo,et al.  On the linear convergence of the alternating direction method of multipliers , 2012, Mathematical Programming.

[53]  V. Weisskopf.  THE INTERNATIONAL INSTITUTE FOR APPLIED SYSTEMS ANALYSIS , 2022 .