Provably Accelerated Randomized Gossip Algorithms

In this work we present novel provably accelerated gossip algorithms for solving the average consensus problem. The proposed protocols are inspired from the recently developed accelerated variants of the randomized Kaczmarz method - a popular method for solving linear systems. In each gossip iteration all nodes of the network update their values but only a pair of them exchange their private information. Numerical experiments on popular wireless sensor networks showing the benefits of our protocols are also presented.

[1]  Stephen J. Wright,et al.  An accelerated randomized Kaczmarz algorithm , 2013, Math. Comput..

[2]  Anand D. Sarwate,et al.  Broadcast Gossip Algorithms for Consensus , 2009, IEEE Transactions on Signal Processing.

[3]  Soummya Kar,et al.  Gossip Algorithms for Distributed Signal Processing , 2010, Proceedings of the IEEE.

[4]  R. Vershynin,et al.  A Randomized Kaczmarz Algorithm with Exponential Convergence , 2007, math/0702226.

[5]  Necdet Serhat Aybat,et al.  Decentralized computation of effective resistances and acceleration of consensus algorithms , 2017, 2017 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[6]  Deanna Needell,et al.  Paved with Good Intentions: Analysis of a Randomized Block Kaczmarz Method , 2012, ArXiv.

[7]  Nikolaos M. Freris,et al.  Randomized Extended Kaczmarz for Solving Least Squares , 2012, SIAM J. Matrix Anal. Appl..

[8]  Alex Olshevsky,et al.  Linear Time Average Consensus on Fixed Graphs and Implications for Decentralized Optimization and Multi-Agent Control , 2014, 1411.4186.

[9]  John N. Tsitsiklis,et al.  Convergence Speed in Distributed Consensus and Averaging , 2009, SIAM J. Control. Optim..

[10]  Robert M. Gower,et al.  Accelerated Stochastic Matrix Inversion: General Theory and Speeding up BFGS Rules for Faster Second-Order Optimization , 2018, NeurIPS.

[11]  Peter Richtárik,et al.  Stochastic Dual Ascent for Solving Linear Systems , 2015, ArXiv.

[12]  M. Degroot.  Reaching a Consensus , 1974 .

[13]  Peter Richtárik,et al.  Privacy preserving randomized gossip algorithms , 2017, 1706.07636.

[14]  Peter Richtárik,et al.  Randomized Iterative Methods for Linear Systems , 2015, SIAM J. Matrix Anal. Appl..

[15]  Peter Richtárik,et al.  Accelerated Gossip via Stochastic Heavy Ball Method , 2018, 2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[16]  Nikolaos M. Freris,et al.  Randomized gossip algorithms for solving Laplacian systems , 2015, 2015 European Control Conference (ECC).

[17]  Deanna Needell,et al.  Convergence Properties of the Randomized Extended Gauss-Seidel and Kaczmarz Methods , 2015, SIAM J. Matrix Anal. Appl..

[18]  D. Needell.  Randomized Kaczmarz solver for noisy linear systems , 2009, 0902.0958.

[19]  Brian D. O. Anderson,et al.  Analysis of accelerated gossip algorithms , 2009, CDC.

[20]  George Cybenko,et al.  Dynamic Load Balancing for Distributed Memory Multiprocessors , 1989, J. Parallel Distributed Comput..

[21]  Yonina C. Eldar,et al.  Acceleration of randomized Kaczmarz method via the Johnson–Lindenstrauss Lemma , 2010, Numerical Algorithms.

[22]  John N. Tsitsiklis,et al.  Distributed Asynchronous Deterministic and Stochastic Gradient Optimization Algorithms , 1984, 1984 American Control Conference.

[23]  Hadrien Hendrikx,et al.  Accelerated Decentralized Optimization with Local Updates for Smooth and Strongly Convex Objectives , 2018, AISTATS.

[24]  Daniel A. Spielman,et al.  Accelerated Gossip Algorithms for Distributed Computation , 2006 .

[25]  Nikolaos M. Freris,et al.  Fast distributed smoothing of relative measurements , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[26]  Alexandros G. Dimakis,et al.  Geographic Gossip: Efficient Averaging for Sensor Networks , 2007, IEEE Transactions on Signal Processing.

[27]  Michael G. Rabbat,et al.  Network Topology and Communication-Computation Tradeoffs in Decentralized Optimization , 2017, Proceedings of the IEEE.

[28]  Michael I. Jordan,et al.  Breaking Locality Accelerates Block Gauss-Seidel , 2017, ICML.

[29]  Yurii Nesterov,et al.  Efficiency of the Accelerated Coordinate Descent Method on Structured Optimization Problems , 2017, SIAM J. Optim..

[30]  Peter Richtárik,et al.  Momentum and stochastic momentum for stochastic gradient, Newton, proximal point and subspace descent methods , 2017, Computational Optimization and Applications.

[31]  Stephen P. Boyd,et al.  Randomized gossip algorithms , 2006, IEEE Transactions on Information Theory.

[32]  D. Needell,et al.  Randomized block Kaczmarz method with projection for solving least squares , 2014, 1403.4192.

[33]  Yurii Nesterov,et al.  Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems , 2012, SIAM J. Optim..

[34]  Christoforos N. Hadjicostis,et al.  Distributed Finite-Time Computation of Digraph Parameters: Left-Eigenvector, Out-Degree and Spectrum , 2016, IEEE Transactions on Control of Network Systems.

[35]  Dirk A. Lorenz,et al.  Linear convergence of the randomized sparse Kaczmarz method , 2016, Mathematical Programming.

[36]  Peter Richtárik,et al.  A new perspective on randomized gossip algorithms , 2016, 2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[37]  Peter Richtárik,et al.  Linearly convergent stochastic heavy ball method for minimizing generalization error , 2017, ArXiv.

[38]  Stephen P. Boyd,et al.  A scheme for robust distributed sensor fusion based on average consensus , 2005, IPSN 2005. Fourth International Symposium on Information Processing in Sensor Networks, 2005..