A Dual Accelerated Method for Online Stochastic Distributed Averaging: From Consensus to Decentralized Policy Evaluation

Motivated by decentralized sensing and policy evaluation problems, we consider a particular type of distributed optimization problem that involves averaging several stochastic, online observations on a network. We design a dual-based method for this consensus problem with Polyak–Ruppert averaging and analyze its behavior. We show that this algorithm attains an accelerated deterministic error depending optimally on the condition number of the network, and also that it has order-optimal stochastic error. This improves on the guarantees of state-of-the-art distributed optimization algorithms when specialized to this setting, and yields—among other things—corollaries for decentralized policy evaluation. Our proofs rely on explicitly studying the evolution of several relevant linear systems, and may be of independent interest. Numerical experiments are provided, which validate our theoretical results and demonstrate that our approach outperforms existing methods in finite-sample scenarios on several natural network topologies.

[1]  Yuxin Chen,et al.  Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reduction , 2020, IEEE Transactions on Information Theory.

[2]  An Accelerated Method For Decentralized Distributed Stochastic Optimization Over Time-Varying Graphs , 2021, 2021 60th IEEE Conference on Decision and Control (CDC).

[3]  Martin J. Wainwright,et al.  Instance-Dependent ℓ∞-Bounds for Policy Evaluation in Tabular Reinforcement Learning , 2021, IEEE Transactions on Information Theory.

[4]  Hadrien Hendrikx,et al.  An Optimal Algorithm for Decentralized Finite Sum Optimization , 2020, SIAM J. Optim..

[5]  Xin Zhang,et al.  Taming Communication and Sample Complexities in Decentralized Policy Evaluation for Cooperative Multi-Agent Reinforcement Learning , 2021, NeurIPS.

[6]  Zhouchen Lin,et al.  Variance Reduced EXTRA and DIGing and Their Optimal Acceleration for Strongly Convex Decentralized Optimization , 2020, J. Mach. Learn. Res..

[7]  Xiang Li,et al.  Distributed Consensus of Heterogeneous Linear Time-Varying Systems on UAVs–USVs Coordination , 2020, IEEE Transactions on Circuits and Systems II: Express Briefs.

[8]  Angelia Nedic,et al.  Distributed Gradient Methods for Convex Machine Learning Problems in Networks: Distributed Optimization , 2020, IEEE Signal Processing Magazine.

[9]  Darina Dvinskikh,et al.  Optimal Decentralized Distributed Algorithms for Stochastic Convex Optimization. , 2019, 1911.07363.

[10]  A. Ozdaglar,et al.  Robust Distributed Accelerated Stochastic Gradient Methods for Multi-Agent Networks , 2019, J. Mach. Learn. Res..

[11]  Tingwen Huang,et al.  Convergence Analysis of a Distributed Optimization Algorithm with a General Unbalanced Directed Communication Network , 2019, IEEE Transactions on Network Science and Engineering.

[12]  Jian Sun,et al.  Adaptive Consensus-Based Distributed Target Tracking With Dynamic Cluster in Sensor Networks , 2019, IEEE Transactions on Cybernetics.

[13]  Thinh T. Doan,et al.  Finite-Time Analysis of Distributed TD(0) with Linear Function Approximation on Multi-Agent Reinforcement Learning , 2019, ICML.

[14]  Haitham Bou-Ammar,et al.  Distributed Newton Method for Large-Scale Consensus Optimization , 2016, IEEE Transactions on Automatic Control.

[15]  S. Kakade,et al.  Reinforcement Learning: Theory and Algorithms , 2019 .

[16]  Ziyang Meng,et al.  A survey of distributed optimization , 2019, Annu. Rev. Control..

[17]  Laurent Massoulié,et al.  Optimal Algorithms for Non-Smooth Distributed Optimization in Networks , 2018, NeurIPS.

[18]  Angelia Nedic,et al.  Distributed Optimization for Control , 2018, Annu. Rev. Control. Robotics Auton. Syst..

[19]  Sonia Martínez,et al.  Distributed Optimal Transport for the Deployment of Swarms , 2018, 2018 IEEE Conference on Decision and Control (CDC).

[20]  Henrik Sandberg,et al.  A Survey of Distributed Optimization and Control Algorithms for Electric Power Systems , 2017, IEEE Transactions on Smart Grid.

[21]  Minyue Fu,et al.  A Distributed Algorithm for Resource Allocation Over Dynamic Digraphs , 2017, IEEE Transactions on Signal Processing.

[22]  Laurent Massoulié,et al.  Optimal Algorithms for Smooth and Strongly Convex Distributed Optimization in Networks , 2017, ICML.

[23]  Martin Guay,et al.  An Extremum-Seeking Controller for Distributed Optimization Over Sensor Networks , 2017, IEEE Transactions on Automatic Control.

[24]  Angelia Nedic,et al.  Convergence Rate of Distributed Averaging Dynamics and Optimization in Networks , 2015, Found. Trends Syst. Control..

[25]  Qing Ling,et al.  EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization , 2014, 1404.6264.

[26]  S. Low,et al.  Distributed algorithm for optimal power flow on a radial network , 2014, 53rd IEEE Conference on Decision and Control.

[27]  H. Vincent Poor,et al.  Distributed Linear Parameter Estimation: Asymptotically Efficient Adaptive Strategies , 2011, SIAM J. Control. Optim..

[28]  Martin J. Wainwright,et al.  Dual Averaging for Distributed Optimization: Convergence Analysis and Network Scaling , 2010, IEEE Transactions on Automatic Control.

[29]  Soummya Kar,et al.  Distributed Parameter Estimation in Sensor Networks: Nonlinear Observation Models and Imperfect Communication , 2008, IEEE Transactions on Information Theory.

[30]  Stefano Giordani,et al.  A Distributed Algorithm for the Multi-Robot Task Allocation Problem , 2010, IEA/AIE.

[31]  Soummya Kar,et al.  Gossip Algorithms for Distributed Signal Processing , 2010, Proceedings of the IEEE.

[32]  Anand D. Sarwate,et al.  Broadcast Gossip Algorithms for Consensus , 2009, IEEE Transactions on Signal Processing.

[33]  Asuman E. Ozdaglar,et al.  Distributed Subgradient Methods for Multi-Agent Optimization , 2009, IEEE Transactions on Automatic Control.

[34]  John N. Tsitsiklis,et al.  Convergence Speed in Distributed Consensus and Averaging , 2009, SIAM J. Control. Optim..

[35]  Stephen P. Boyd,et al.  Distributed average consensus with least-mean-square deviation , 2007, J. Parallel Distributed Comput..

[36]  Stephen P. Boyd,et al.  Randomized gossip algorithms , 2006, IEEE Transactions on Information Theory.

[37]  Stephen P. Boyd,et al.  A scheme for robust distributed sensor fusion based on average consensus , 2005, IPSN 2005. Fourth International Symposium on Information Processing in Sensor Networks, 2005..

[38]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[39]  Stephen P. Boyd,et al.  Fast linear iterations for distributed averaging , 2003, 42nd IEEE International Conference on Decision and Control (IEEE Cat. No.03CH37475).

[40]  Michael Kearns,et al.  Finite-Sample Convergence Rates for Q-Learning and Indirect Algorithms , 1998, NIPS.

[41]  Boris Polyak,et al.  Acceleration of stochastic approximation by averaging , 1992 .

[42]  Seif Haridi,et al.  Distributed Algorithms , 1992, Lecture Notes in Computer Science.

[43]  John N. Tsitsiklis,et al.  Problems in decentralized decision making and computation , 1984 .

[44]  John N. Tsitsiklis,et al.  Distributed Asynchronous Deterministic and Stochastic Gradient Optimization Algorithms , 1984, 1984 American Control Conference.

[45]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .