Expander graph and communication-efficient decentralized optimization

In this paper, we discuss how to design the graph topology to reduce the communication complexity of certain algorithms for decentralized optimization. Our goal is to minimize the total communication needed to achieve a prescribed accuracy. We discover that the so-called expander graphs are near-optimal choices. We propose three approaches to construct expander graphs for different numbers of nodes and node degrees. Our numerical results show that the performance of decentralized optimization is significantly better on expander graphs than other regular graphs.

[1]  Qing Ling,et al.  A proximal gradient algorithm for decentralized nondifferentiable optimization , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2]  J. Dodziuk.  Difference equations, isoperimetric inequality and transience of certain random walks , 1984 .

[3]  Ali H. Sayed,et al.  Diffusion Adaptation Over Networks Under Imperfect Information Exchange and Non-Stationary Data , 2011, IEEE Transactions on Signal Processing.

[4]  N. Linial,et al.  Expander Graphs and their Applications , 2006 .

[5]  Georgios B. Giannakis,et al.  Distributed Spectrum Sensing for Cognitive Radio Networks by Exploiting Sparsity , 2010, IEEE Transactions on Signal Processing.

[6]  Alexander Lubotzky,et al.  Explicit expanders and the Ramanujan conjectures , 1986, STOC '86.

[7]  Gonzalo Mateos,et al.  Group-Lasso on Splines for Spectrum Cartography , 2010, IEEE Transactions on Signal Processing.

[8]  Stephen P. Boyd,et al.  Fast linear iterations for distributed averaging , 2003, 42nd IEEE International Conference on Decision and Control (IEEE Cat. No.03CH37475).

[9]  Qing Ling,et al.  On the Linear Convergence of the ADMM in Decentralized Consensus Optimization , 2013, IEEE Transactions on Signal Processing.

[10]  Soummya Kar,et al.  Gossip Algorithms for Distributed Signal Processing , 2010, Proceedings of the IEEE.

[11]  M. Murty.  Ramanujan Graphs , 1965 .

[12]  João M. F. Xavier,et al.  D-ADMM: A Communication-Efficient Distributed Algorithm for Separable Optimization , 2012, IEEE Transactions on Signal Processing.

[13]  Noga Alon,et al.  The Probabilistic Method , 2015, Fundamentals of Ramsey Theory.

[14]  John N. Tsitsiklis,et al.  Problems in decentralized decision making and computation , 1984 .

[15]  R. Elzinga.  Producing the graphs of Lubotzky Phillips and Sarnak in Matlab , 2010 .

[16]  Qing Ling,et al.  Decentralized low-rank matrix completion , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[17]  Stephen P. Boyd,et al.  Distributed average consensus with least-mean-square deviation , 2007, J. Parallel Distributed Comput..

[18]  Stephen P. Boyd,et al.  Growing Well-connected Graphs , 2006, Proceedings of the 45th IEEE Conference on Decision and Control.

[19]  Qing Ling,et al.  Decentralized Sparse Signal Recovery for Compressive Sleeping Wireless Sensor Networks , 2010, IEEE Transactions on Signal Processing.

[20]  Gonzalo Mateos,et al.  Distributed Sparse Linear Regression , 2010, IEEE Transactions on Signal Processing.

[21]  Qing Ling,et al.  Decentralized Jointly Sparse Optimization by Reweighted Minimization , 2013 .

[22]  A. L. O N.  On the edge-expansion of graphs , 2002 .

[23]  Euhanna Ghadimi,et al.  The ADMM algorithm for distributed averaging: Convergence rates and optimal parameter selection , 2014, 2014 48th Asilomar Conference on Signals, Systems and Computers.

[24]  Björn Johansson,et al.  On Distributed Optimization in Networked Systems , 2008 .

[25]  Qing Ling,et al.  On the Convergence of Decentralized Gradient Descent , 2013, SIAM J. Optim..

[26]  H. Vincent Poor,et al.  A Collaborative Training Algorithm for Distributed Learning , 2009, IEEE Transactions on Information Theory.

[27]  Noga Alon,et al.  Semi-direct product in groups and zig-zag product in graphs: connections and applications , 2001, Proceedings 2001 IEEE International Conference on Cluster Computing.

[28]  Qing Ling,et al.  Decentralized Jointly Sparse Optimization by Reweighted $\ell_{q}$ Minimization , 2013, IEEE Transactions on Signal Processing.

[29]  Mehran Mesbahi,et al.  On maximizing the second smallest eigenvalue of a state-dependent graph Laplacian , 2006, IEEE Transactions on Automatic Control.

[30]  Qing Ling,et al.  A linearized bregman algorithm for decentralized basis pursuit , 2013, 21st European Signal Processing Conference (EUSIPCO 2013).

[31]  Michael William Newman,et al.  The Laplacian spectrum of graphs , 2001 .

[32]  Stephen P. Boyd,et al.  Fastest Mixing Markov Chain on a Graph , 2004, SIAM Rev..

[33]  Nikhil Srivastava,et al.  Interlacing Families I: Bipartite Ramanujan Graphs of All Degrees , 2013, 2013 IEEE 54th Annual Symposium on Foundations of Computer Science.

[34]  Georgios B. Giannakis,et al.  Distributed Robust Power System State Estimation , 2012, IEEE Transactions on Power Systems.

[35]  Georgios B. Giannakis,et al.  Consensus-Based Distributed Support Vector Machines , 2010, J. Mach. Learn. Res..

[36]  Qing Ling,et al.  EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization , 2014, 1404.6264.

[37]  Annie I-An Chen,et al.  Fast Distributed First-Order Methods , 2012 .

[38]  Joel Friedman,et al.  A proof of Alon's second eigenvalue conjecture and related problems , 2004, ArXiv.