A Dual Accelerated Method for a Class of Distributed Optimization Problems: From Consensus to Decentralized Policy Evaluation

Motivated by decentralized sensing and policy evaluation problems, we consider a particular type of distributed stochastic optimization problem over a network, called the online stochastic distributed averaging problem. We design a dual-based method for this distributed consensus problem with Polyak–Ruppert averaging and analyze its behavior. We show that the proposed algorithm attains an accelerated deterministic error depending optimally on the condition number of the network, and also that it has an order-optimal stochastic error. This improves on the guarantees of state-of-the-art distributed stochastic optimization algorithms when specialized to this setting, and yields—among other things—corollaries for decentralized policy evaluation. Our proofs rely on explicitly studying the evolution of several relevant linear systems, and may be of independent interest.

[1]  J. Romberg,et al.  A Dual Accelerated Method for Online Stochastic Distributed Averaging: From Consensus to Decentralized Policy Evaluation , 2022, ArXiv.

[2]  P. Dvurechensky,et al.  An Accelerated Method For Decentralized Distributed Stochastic Optimization Over Time-Varying Graphs , 2021, 2021 60th IEEE Conference on Decision and Control (CDC).

[3]  Martin J. Wainwright,et al.  Instance-Dependent ℓ∞-Bounds for Policy Evaluation in Tabular Reinforcement Learning , 2021, IEEE Transactions on Information Theory.

[4]  Eduard A. Gorbunov,et al.  Recent theoretical advances in decentralized distributed convex optimization. , 2020, 2011.13259.

[5]  Zhouchen Lin,et al.  Variance Reduced EXTRA and DIGing and Their Optimal Acceleration for Strongly Convex Decentralized Optimization , 2020, J. Mach. Learn. Res..

[6]  Xiang Li,et al.  Distributed Consensus of Heterogeneous Linear Time-Varying Systems on UAVs–USVs Coordination , 2020, IEEE Transactions on Circuits and Systems II: Express Briefs.

[7]  Hadrien Hendrikx,et al.  An Optimal Algorithm for Decentralized Finite Sum Optimization , 2020, SIAM J. Optim..

[8]  Angelia Nedic,et al.  Distributed Gradient Methods for Convex Machine Learning Problems in Networks: Distributed Optimization , 2020, IEEE Signal Processing Magazine.

[9]  Darina Dvinskikh,et al.  Optimal Decentralized Distributed Algorithms for Stochastic Convex Optimization. , 2019, 1911.07363.

[10]  A. Ozdaglar,et al.  Robust Distributed Accelerated Stochastic Gradient Methods for Multi-Agent Networks , 2019, J. Mach. Learn. Res..

[11]  Tingwen Huang,et al.  Convergence Analysis of a Distributed Optimization Algorithm with a General Unbalanced Directed Communication Network , 2019, IEEE Transactions on Network Science and Engineering.

[12]  Jian Sun,et al.  Adaptive Consensus-Based Distributed Target Tracking With Dynamic Cluster in Sensor Networks , 2019, IEEE Transactions on Cybernetics.

[13]  Thinh T. Doan,et al.  Finite-Time Analysis of Distributed TD(0) with Linear Function Approximation on Multi-Agent Reinforcement Learning , 2019, ICML.

[14]  Laurent Massoulié,et al.  Optimal Algorithms for Non-Smooth Distributed Optimization in Networks , 2018, NeurIPS.

[15]  Angelia Nedic,et al.  Distributed Optimization for Control , 2018, Annu. Rev. Control. Robotics Auton. Syst..

[16]  Henrik Sandberg,et al.  A Survey of Distributed Optimization and Control Algorithms for Electric Power Systems , 2017, IEEE Transactions on Smart Grid.

[17]  Laurent Massoulié,et al.  Optimal Algorithms for Smooth and Strongly Convex Distributed Optimization in Networks , 2017, ICML.

[18]  Martin Guay,et al.  An Extremum-Seeking Controller for Distributed Optimization Over Sensor Networks , 2017, IEEE Transactions on Automatic Control.

[19]  Haitham Bou-Ammar,et al.  Distributed Newton Method for Large-Scale Consensus Optimization , 2016, IEEE Transactions on Automatic Control.

[20]  Angelia Nedic,et al.  Convergence Rate of Distributed Averaging Dynamics and Optimization in Networks , 2015, Found. Trends Syst. Control..

[21]  Qing Ling,et al.  EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization , 2014, 1404.6264.

[22]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[23]  S. Low,et al.  Distributed algorithm for optimal power flow on a radial network , 2014, 53rd IEEE Conference on Decision and Control.

[24]  H. Vincent Poor,et al.  Distributed Linear Parameter Estimation: Asymptotically Efficient Adaptive Strategies , 2011, SIAM J. Control. Optim..

[25]  Stefano Giordani,et al.  A Distributed Algorithm for the Multi-Robot Task Allocation Problem , 2010, IEA/AIE.

[26]  Soummya Kar,et al.  Gossip Algorithms for Distributed Signal Processing , 2010, Proceedings of the IEEE.

[27]  Anand D. Sarwate,et al.  Broadcast Gossip Algorithms for Consensus , 2009, IEEE Transactions on Signal Processing.

[28]  Asuman E. Ozdaglar,et al.  Distributed Subgradient Methods for Multi-Agent Optimization , 2009, IEEE Transactions on Automatic Control.

[29]  Soummya Kar,et al.  Distributed Parameter Estimation in Sensor Networks: Nonlinear Observation Models and Imperfect Communication , 2008, IEEE Transactions on Information Theory.

[30]  John N. Tsitsiklis,et al.  Convergence Speed in Distributed Consensus and Averaging , 2009, SIAM J. Control. Optim..

[31]  Stephen P. Boyd,et al.  Randomized gossip algorithms , 2006, IEEE Transactions on Information Theory.

[32]  Stephen P. Boyd,et al.  A scheme for robust distributed sensor fusion based on average consensus , 2005, IPSN 2005. Fourth International Symposium on Information Processing in Sensor Networks, 2005..

[33]  Stephen P. Boyd,et al.  Fast linear iterations for distributed averaging , 2003, 42nd IEEE International Conference on Decision and Control (IEEE Cat. No.03CH37475).

[34]  Michael Kearns,et al.  Finite-Sample Convergence Rates for Q-Learning and Indirect Algorithms , 1998, NIPS.

[35]  Boris Polyak,et al.  Acceleration of stochastic approximation by averaging , 1992 .

[36]  John N. Tsitsiklis,et al.  Problems in decentralized decision making and computation , 1984 .

[37]  John N. Tsitsiklis,et al.  Distributed Asynchronous Deterministic and Stochastic Gradient Optimization Algorithms , 1984, 1984 American Control Conference.

[38]  Xin Zhang,et al.  Taming Communication and Sample Complexities in Decentralized Policy Evaluation for Cooperative Multi-Agent Reinforcement Learning , 2021, NeurIPS.

[39]  Ziyang Meng,et al.  A survey of distributed optimization , 2019, Annu. Rev. Control..

[40]  Stephen P. Boyd,et al.  Distributed average consensus with least-mean-square deviation , 2007, J. Parallel Distributed Comput..

[41]  Seif Haridi,et al.  Distributed Algorithms , 1992, Lecture Notes in Computer Science.

[42]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .