Sample Efficient Decentralized Stochastic Frank-Wolfe Methods for Continuous DR-Submodular Maximization

Continuous DR-submodular maximization is an important machine learning problem, which covers numerous popular applications. With the emergence of large-scale distributed data, developing efficient algorithms for the continuous DRsubmodular maximization, such as the decentralized Frank-Wolfe method, became an important challenge. However, existing decentralized FrankWolfe methods for this kind of problem have the sample complexity of O(1/ ), incurring a large computational overhead. In this paper, we propose two novel sample efficient decentralized FrankWolfe methods to address this challenge. Our theoretical results demonstrate that the sample complexity of the two proposed methods is O(1/ ), which is better than O(1/ ) of the existing methods. As far as we know, this is the first published result achieving such a favorable sample complexity. Extensive experimental results confirm the effectiveness of the proposed methods.

[1]  Tong Zhang,et al.  SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator , 2018, NeurIPS.

[2]  Anna Scaglione,et al.  Decentralized Frank–Wolfe Algorithm for Convex and Nonconvex Problems , 2016, IEEE Transactions on Automatic Control.

[3]  Amin Karbasi,et al.  Online Continuous Submodular Maximization , 2018, AISTATS.

[4]  Angelia Nedic,et al.  Distributed stochastic gradient tracking methods , 2018, Mathematical Programming.

[5]  Songtao Lu,et al.  GNSD: a Gradient-Tracking Based Nonconvex Stochastic Algorithm for Decentralized Optimization , 2019, 2019 IEEE Data Science Workshop (DSW).

[6]  Rong Jin,et al.  On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization , 2019, ICML.

[7]  Haoran Sun,et al.  Improving the Sample and Communication Complexity for Decentralized Non-Convex Optimization: A Joint Gradient Estimation and Tracking Approach , 2019, ArXiv.

[8]  Amin Karbasi,et al.  One Sample Stochastic Frank-Wolfe , 2019, AISTATS.

[9]  Francesco Orabona,et al.  Momentum-Based Variance Reduction in Non-Convex SGD , 2019, NeurIPS.

[10]  Andreas Krause,et al.  Guaranteed Non-convex Optimization: Submodular Maximization over Continuous Domains , 2016, AISTATS.

[11]  Wei Zhang,et al.  Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent , 2017, NIPS.

[12]  Amin Karbasi,et al.  Gradient Methods for Submodular Maximization , 2017, NIPS.

[13]  Maryam Fazel,et al.  Designing smoothing functions for improved worst-case competitive ratio in online optimization , 2016, NIPS.

[14]  Stefanie Jegelka,et al.  Robust Budget Allocation Via Continuous Submodular Functions , 2017, Applied Mathematics & Optimization.

[15]  Martin Jaggi,et al.  A Unified Theory of Decentralized SGD with Changing Topology and Local Updates , 2020, ICML.

[16]  Zebang Shen,et al.  Decentralized Gradient Tracking for Continuous DR-Submodular Maximization , 2019, AISTATS.

[17]  Amin Karbasi,et al.  A Submodular Approach to Create Individualized Parcellations of the Human Brain , 2017, MICCAI.

[18]  Heng Huang,et al.  Periodic Stochastic Gradient Descent with Momentum for Decentralized Training , 2020, ArXiv.

[19]  Martin Jaggi,et al.  Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication , 2019, ICML.

[20]  U. Khan,et al.  A near-optimal stochastic gradient method for decentralized non-convex finite-sum optimization , 2020, ArXiv.

[21]  Amin Karbasi,et al.  Stochastic Conditional Gradient Methods: From Convex Minimization to Submodular Maximization , 2018, J. Mach. Learn. Res..

[22]  Amin Karbasi,et al.  Stochastic Conditional Gradient++ , 2019, SIAM J. Optim..

[23]  Amin Karbasi,et al.  Decentralized Submodular Maximization: Bridging Discrete and Continuous Settings , 2018, ICML.

[24]  C. Guestrin,et al.  Near-optimal sensor placements: maximizing information while minimizing communication cost , 2006, 2006 5th International Conference on Information Processing in Sensor Networks.