FAST ASYNCHRONOUS DECENTRALIZED OPTIMIZATION: ALLOWING MULTIPLE MASTERS

The present paper deals with asynchronous decentralized optimization over networks. Pertinent algorithms are either centralized relying on a specific topology, where a single master connects all workers, or decentralized devoid of any master by only exchanging information between single-hop neighbors. The present work bridges the gap of existing approaches with a novel hybrid framework that is capable of accommodating multiple masters. Moreover, it enables considerable acceleration of decentralized approaches without physically deploying masters, thus making it possible to achieve a desirable tradeoff between convergence and communication/computation complexity by tuning the configuration. Numerical tests showcase advantages over decentralized counterparts.

[1]  Mingyi Hong,et al.  A Distributed, Asynchronous, and Incremental Algorithm for Nonconvex Optimization: An ADMM Approach , 2014, IEEE Transactions on Control of Network Systems.

[2]  Xiangfeng Wang,et al.  Asynchronous Distributed ADMM for Large-Scale Optimization—Part I: Algorithm and Convergence Analysis , 2015, IEEE Transactions on Signal Processing.

[3]  Ming Yan,et al.  ARock: an Algorithmic Framework for Asynchronous Parallel Coordinate Updates , 2015, SIAM J. Sci. Comput..

[4]  Pascal Bianchi,et al.  Asynchronous distributed optimization using a randomized alternating direction method of multipliers , 2013, 52nd IEEE Conference on Decision and Control.

[5]  Qing Ling,et al.  On the Linear Convergence of the ADMM in Decentralized Consensus Optimization , 2013, IEEE Transactions on Signal Processing.

[6]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[7]  Qing Ling,et al.  Decentralized learning for wireless communications and networking , 2015, ArXiv.

[8]  Asuman E. Ozdaglar,et al.  On the O(1=k) convergence of asynchronous distributed alternating Direction Method of Multipliers , 2013, 2013 IEEE Global Conference on Signal and Information Processing.

[9]  Ali H. Sayed,et al.  Decentralized Consensus Optimization With Asynchrony and Delays , 2016, IEEE Transactions on Signal and Information Processing over Networks.

[10]  Georgios B. Giannakis,et al.  Fast Decentralized Optimization over Networks , 2018, ArXiv.

[11]  Reza Olfati-Saber,et al.  Consensus and Cooperation in Networked Multi-Agent Systems , 2007, Proceedings of the IEEE.

[12]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[13]  Pascal Bianchi,et al.  Explicit Convergence Rate of a Distributed Alternating Direction Method of Multipliers , 2013, IEEE Transactions on Automatic Control.

[14]  James T. Kwok,et al.  Asynchronous Distributed ADMM for Consensus Optimization , 2014, ICML.

[15]  Stephen P. Boyd,et al.  Graph Implementations for Nonsmooth Convex Programs , 2008, Recent Advances in Learning and Control.

[16]  Jarvis D. Haupt,et al.  Communication-Efficient distributed optimization for sparse learning via two-way truncation , 2017, 2017 IEEE 7th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP).