A fast-convergence distributed support vector machine in small-scale strongly connected networks

In this paper, a fast-convergence distributed support vector machine (FDSVM) algorithm is proposed, aiming at efficiently solving the problem of distributed SVM training. Rather than exchanging information only among immediate neighbor sites, the proposed FDSVM employs a deterministic gossip protocol-based communication policy to accelerate diffusing information around the network, in which each site communicates with others in a flooding and iterative manner. This communication policy significantly reduces the total number of iterations, thus further speeding up the convergence of the algorithm. In addition, the proposed algorithm is proved to converge to the global optimum in finite steps over an arbitrary strongly connected network (SCN). Experiments on various benchmark data sets show that the proposed FDSVM consistently outperforms the related state-of-the-art approach for most networks, especially in the ring network, in terms of the total training time.

[1]  S. Sathiya Keerthi,et al.  Improvements to Platt's SMO Algorithm for SVM Classifier Design , 2001, Neural Computation.

[2]  Marcus Brunner,et al.  Self-Managing Distributed Systems , 2003, Lecture Notes in Computer Science.

[3]  Thorsten Joachims,et al.  Making large-scale support vector machine learning practical , 1999 .

[4]  Huan Liu,et al.  Handling concept drifts in incremental learning with support vector machines , 1999, KDD '99.

[5]  S. Sathiya Keerthi,et al.  Developing parallel sequential minimal optimization for fast training support vector machine , 2006, Neurocomputing.

[6]  Igor Durdanovic,et al.  Parallel Support Vector Machines: The Cascade SVM , 2004, NIPS.

[7]  Lei Wang,et al.  A parallel training algorithm of support vector machines based on the MTC architecture , 2008, 2008 IEEE Conference on Cybernetics and Intelligent Systems.

[8]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[9]  Youn Jung Park,et al.  SVM-based feature extraction for face recognition , 2010, Pattern Recognit..

[11]  Vwani P. Roychowdhury,et al.  Distributed Parallel Support Vector Machines in Strongly Connected Networks , 2008, IEEE Transactions on Neural Networks.

[12]  Gopal Pandurangan,et al.  Optimal gossip-based aggregate computation , 2010, SPAA '10.

[13]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[14]  Reshma Khemchandani,et al.  Fast and robust learning through fuzzy linear proximal support vector machines , 2004, Neurocomputing.

[15]  Andrew S. Tanenbaum,et al.  Distributed systems: Principles and Paradigms , 2001 .

[16]  Devavrat Shah,et al.  Gossip Algorithms , 2009, Found. Trends Netw..

[17]  Maarten van Steen,et al.  An Epidemic Protocol for Managing Routing Tables in Very Large Peer-to-Peer Networks , 2003, DSOM.

[18]  Hsin-Chang Yang,et al.  Construction of supervised and unsupervised learning systems for multilingual text categorization , 2009, Expert Syst. Appl..

[19]  Yan Zhou,et al.  A scalable support vector machine for distributed classification in ad hoc sensor networks , 2010, Neurocomputing.

[20]  David Liben-Nowell.  Gossip is synteny: incomplete gossip and an exact algorithm for syntenic distance , 2001, SODA '01.

[21]  Federico Girosi,et al.  Training support vector machines: an application to face detection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[22]  Jian-xiong Dong,et al.  A Fast Parallel Optimization for Training Support Vector Machine , 2003, MLDM.

[23]  Luca Zanni,et al.  A parallel solver for large quadratic programs in training support vector machines , 2003, Parallel Comput..

[24]  Yixue Li,et al.  Predicting rRNA-, RNA-, and DNA-binding proteins from primary structure with support vector machines. , 2006, Journal of theoretical biology.