A Second-Order Proximal Algorithm for Consensus Optimization

We develop a distributed second-order proximal algorithm, referred to as SoPro, to address in-network consensus optimization. The proposed SoPro algorithm converges linearly to the exact optimal solution, provided that the global cost function is locally restricted strongly convex. This relaxes the standard global strong convexity condition required by the existing distributed optimization algorithms to establish linear convergence. In addition, we demonstrate that SoPro is computation- and communication-efficient in comparison with the state-of-the-art distributed second-order methods. Finally, extensive simulations illustrate the competitive convergence performance of SoPro.

[1]  Marc Teboulle,et al.  An $O(1/k)$ Gradient Method for Network Resource Allocation Problems , 2014, IEEE Transactions on Control of Network Systems.

[2]  Na Li,et al.  Harnessing smoothness to accelerate distributed optimization , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[3]  Van Sy Mai,et al.  Linear Convergence in Optimization Over Directed Graphs With Row-Stochastic Matrices , 2016, IEEE Transactions on Automatic Control.

[4]  Qing Ling,et al.  EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization , 2014, 1404.6264.

[5]  Dusan Jakovetic,et al.  A Unification and Generalization of Exact Distributed First-Order Methods , 2017, IEEE Transactions on Signal and Information Processing over Networks.

[6]  Dragana Bajović,et al.  Newton-like Method with Diagonal Correction for Distributed Optimization , 2015, SIAM J. Optim..

[7]  Dongyan Xu,et al.  Robust computation of aggregates in wireless sensor networks: distributed randomized algorithms and analysis , 2005, IPSN 2005. Fourth International Symposium on Information Processing in Sensor Networks, 2005..

[8]  Michael William Newman,et al.  The Laplacian spectrum of graphs , 2001 .

[9]  Guang-Hong Yang,et al.  Augmented Lagrange algorithms for distributed optimization over multi-agent networks via edge-based method , 2018, Autom..

[10]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[11]  Usman A. Khan,et al.  A Linear Algorithm for Optimization Over Directed Graphs With Geometric Convergence , 2018, IEEE Control Systems Letters.

[12]  Asuman E. Ozdaglar,et al.  Constrained Consensus and Optimization in Multi-Agent Networks , 2008, IEEE Transactions on Automatic Control.

[13]  Mung Chiang,et al.  The value of clustering in distributed estimation for sensor networks , 2005, 2005 International Conference on Wireless Networks, Communications and Mobile Computing.

[14]  Aryan Mokhtari,et al.  Decentralized Quasi-Newton Methods , 2016, IEEE Transactions on Signal Processing.

[15]  Asuman E. Ozdaglar,et al.  Convergence Rate of Distributed ADMM Over Networks , 2016, IEEE Transactions on Automatic Control.

[16]  Na Li,et al.  Accelerated Distributed Nesterov Gradient Descent , 2017, IEEE Transactions on Automatic Control.

[17]  Francis R. Bach,et al.  Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regression , 2013, J. Mach. Learn. Res..

[18]  José M. F. Moura,et al.  Fast Distributed Gradient Methods , 2011, IEEE Transactions on Automatic Control.

[19]  Usman A. Khan,et al.  DEXTRA: A Fast Algorithm for Optimization Over Directed Graphs , 2017, IEEE Transactions on Automatic Control.

[20]  Qing Ling,et al.  A Proximal Gradient Algorithm for Decentralized Composite Optimization , 2015, IEEE Transactions on Signal Processing.

[21]  Aryan Mokhtari,et al.  Network Newton Distributed Optimization Methods , 2017, IEEE Transactions on Signal Processing.

[22]  Angelia Nedic,et al.  Stochastic Gradient-Push for Strongly Convex Functions on Time-Varying Directed Graphs , 2014, IEEE Transactions on Automatic Control.

[23]  Usman A. Khan,et al.  ADD-OPT: Accelerated Distributed Directed Optimization , 2016, IEEE Transactions on Automatic Control.

[24]  Angelia Nedic,et al.  Distributed optimization over time-varying directed graphs , 2013, 52nd IEEE Conference on Decision and Control.

[25]  Aryan Mokhtari,et al.  A Decentralized Second-Order Method with Exact Linear Convergence Rate for Consensus Optimization , 2016, IEEE Transactions on Signal and Information Processing over Networks.

[26]  Pascal Bianchi,et al.  A Coordinate Descent Primal-Dual Algorithm and Application to Distributed Asynchronous Optimization , 2014, IEEE Transactions on Automatic Control.

[27]  Damiano Varagnolo,et al.  Newton-Raphson Consensus for Distributed Convex Optimization , 2015, IEEE Transactions on Automatic Control.

[28]  Aryan Mokhtari,et al.  DQM: Decentralized Quadratically Approximated Alternating Direction Method of Multipliers , 2016, IEEE Transactions on Signal Processing.