Distributed Support Vector Machines Over Dynamic Balanced Directed Networks

In this letter, we consider the binary classification problem via distributed Support Vector Machines (SVMs), where the idea is to train a network of agents, with limited share of data, to cooperatively learn the SVM classifier for the global database. Agents only share processed information regarding the classifier parameters and the gradient of the local loss functions instead of their raw data. In contrast to the existing work, we propose a continuous-time algorithm that incorporates network topology changes in discrete jumps. This hybrid nature allows us to remove chattering that arises because of the discretization of the underlying CT process. We show that the proposed algorithm converges to the SVM classifier over time-varying weight balanced directed graphs by using arguments from the matrix perturbation theory.

[1]  Jason Weston,et al.  Fast Kernel Classifiers with Online and Active Learning , 2005, J. Mach. Learn. Res..

[2]  Wei Ren,et al.  Distributed Continuous-Time Convex Optimization With Time-Varying Cost Functions , 2017, IEEE Transactions on Automatic Control.

[3]  Bahman Gharesifard,et al.  Distributed Continuous-Time Convex Optimization on Weight-Balanced Digraphs , 2012, IEEE Transactions on Automatic Control.

[4]  Alfred O. Hero,et al.  Fixed-time Distributed Optimization under Time-Varying Communication Topology , 2019 .

[5]  Hod Lipson,et al.  Learning symbolic representations of hybrid dynamical systems , 2012, J. Mach. Learn. Res..

[6]  Emilio Parrado-Hernández,et al.  Distributed support vector machines , 2006, IEEE Trans. Neural Networks.

[7]  Riadh Omheni,et al.  A globally and quadratically convergent primal–dual augmented Lagrangian algorithm for equality constrained optimization , 2017, Optim. Methods Softw..

[8]  Olivier Chapelle,et al.  Training a Support Vector Machine in the Primal , 2007, Neural Computation.

[9]  Richard M. Murray,et al.  Consensus problems in networks of agents with switching topology and time-delays , 2004, IEEE Transactions on Automatic Control.

[10]  Jorge Cortés,et al.  Distributed Algorithm via Continuously Differentiable Exact Penalty Method for Network Optimization , 2018, 2018 IEEE Conference on Decision and Control (CDC).

[11]  Ermin Wei,et al.  A Fast Distributed Asynchronous Newton-Based Optimization Algorithm , 2019, IEEE Transactions on Automatic Control.

[12]  Usman A. Khan,et al.  A General Framework for Decentralized Optimization With First-Order Methods , 2020, Proceedings of the IEEE.

[13]  V. N. Bogaevski,et al.  Matrix Perturbation Theory , 1991 .

[14]  Qing-Long Han,et al.  Distributed Optimization for Multiagent Systems: An Edge-Based Fixed-Time Consensus Approach , 2019, IEEE Transactions on Cybernetics.

[15]  Zhengtao Ding,et al.  Time-varying multi-objective optimisation over switching graphs via fixed-time consensus algorithms , 2020, Int. J. Syst. Sci..

[16]  R. Sanfelice,et al.  Hybrid dynamical systems , 2009, IEEE Control Systems.

[17]  Georgios B. Giannakis,et al.  Time-Varying Convex Optimization: Time-Structured Algorithms and Applications , 2020, Proceedings of the IEEE.

[18]  Mohammadreza Doostmohammadian,et al.  Single-Bit Consensus With Finite-Time Convergence: Theory and Applications , 2020, IEEE Transactions on Aerospace and Electronic Systems.

[19]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[20]  Vwani P. Roychowdhury,et al.  Distributed Parallel Support Vector Machines in Strongly Connected Networks , 2008, IEEE Transactions on Neural Networks.

[21]  Georgios B. Giannakis,et al.  Consensus-Based Distributed Support Vector Machines , 2010, J. Mach. Learn. Res..

[22]  Hao Wang,et al.  PSVM : Parallelizing Support Vector Machines on Distributed Computers , 2007 .

[23]  Soummya Kar,et al.  Decentralized Stochastic Optimization and Machine Learning: A Unified Variance-Reduction Framework for Robust Performance and Fast Convergence , 2020, IEEE Signal Processing Magazine.

[24]  Qing Ling,et al.  On the Convergence of Decentralized Gradient Descent , 2013, SIAM J. Optim..

[25]  Kai Cai,et al.  Average consensus on general strongly connected digraphs , 2012, Autom..