Parallel Computing of Support Vector Machines

The immense amount of data created by digitalization requires parallel computing for machine-learning methods. While there are many parallel implementations for support vector machines (SVMs), there is no clear suggestion for every application scenario. Many factor—including optimization algorithm, problem size and dimension, kernel function, parallel programming stack, and hardware architecture—impact the efficiency of implementations. It is up to the user to balance trade-offs, particularly between computation time and classification accuracy. In this survey, we review the state-of-the-art implementations of SVMs, their pros and cons, and suggest possible avenues for future research.

[1]  Luca Zanni,et al.  Gradient projection methods for quadratic programs and applications in training support vector machines , 2005, Optim. Methods Softw..

[2]  Bruno Lang,et al.  Efficient Implementation of Serial and Parallel Support Vector Machine Training with a Multi-Parameter Kernel for Large-Scale Data , 2007 .

[3]  Thanh-Nghi Do,et al.  Non-linear Classification of Massive Datasets with a Parallel Algorithm of Local Support Vector Machines , 2015, ICCSAMA.

[4]  François Poulet,et al.  Classifying one billion data with a new distributed svm algorithm , 2006, 2006 International Conference onResearch, Innovation and Vision for the Future.

[5]  Edward Y. Chang,et al.  Parallelizing Support Vector Machines on Distributed Computers , 2007, NIPS.

[6]  Kurt Keutzer,et al.  Fast support vector machine training and classification on graphics processors , 2008, ICML '08.

[7]  Krzysztof Sopyla,et al.  SVM with CUDA Accelerated Kernels for Big Sparse Problems , 2012, ICAISC.

[8]  Joshua D. Griffin,et al.  Using an iterative linear solver in an interior-point method for generating support vector machines , 2010, Comput. Optim. Appl..

[9]  Christos-Savvas Bouganis,et al.  Performance comparison of GPU and FPGA architectures for the SVM training problem , 2009, 2009 International Conference on Field-Programmable Technology.

[10]  Qi Li,et al.  Fast parallel machine learning algorithms for large datasets using graphic processing unit , 2011 .

[11]  Patrice Y. Simard,et al.  Using GPUs for machine learning algorithms , 2005, Eighth International Conference on Document Analysis and Recognition (ICDAR'05).

[12]  Abdulsalam Yassine,et al.  Cloud-based SVM for food categorization , 2015, Multimedia tools and applications.

[13]  Alexander J. Smola,et al.  Parallelized Stochastic Gradient Descent , 2010, NIPS.

[14]  Wen-mei W. Hwu,et al.  GPU Computing Gems Emerald Edition , 2011 .

[15]  Ji-Bo Wang,et al.  GPU Accelerated Support Vector Machines for Mining High-Throughput Screening Data , 2009, J. Chem. Inf. Model..

[16]  Stefan Rüping,et al.  Incremental Learning with Support Vector Machines , 2001, ICDM.

[17]  Maozhen Li,et al.  Parallelizing multiclass support vector machines for scalable image annotation , 2011, 2011 Eighth International Conference on Fuzzy Systems and Knowledge Discovery (FSKD).

[18]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[19]  François Poulet,et al.  Speed Up SVM Algorithm for Massive Classification Tasks , 2008, ADMA.

[20]  Chee Kheong Siew,et al.  Fast Modular network implementation for support vector machines , 2005, IEEE Transactions on Neural Networks.

[21]  Georgios B. Giannakis,et al.  Consensus-Based Distributed Support Vector Machines , 2010, J. Mach. Learn. Res..

[22]  Le Song,et al.  CA-SVM: Communication-Avoiding Support Vector Machines on Distributed Systems , 2015, 2015 IEEE International Parallel and Distributed Processing Symposium.

[23]  Fabien Lauer,et al.  Efficient Optimization of Multi-class Support Vector Machines with MSVMpack , 2015, MCO.

[24]  Jian-xiong Dong,et al.  Fast SVM training algorithm with decomposition on very large data sets , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Ioannis Kompatsiaris,et al.  GPU acceleration for support vector machines , 2011, WIAMIS 2011.

[26]  Jeyanthi Narasimhan,et al.  Fast Support Vector Machines Using Parallel Adaptive Shrinking on Distributed Systems , 2014, ArXiv.

[27]  Kuruvilla Varghese,et al.  Hybrid Working Set Algorithm for SVM Learning With a Kernel Coprocessor on FPGA , 2015, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[28]  Padma Raghavan,et al.  Characterizing sparse preconditioner performance for the support vector machine kernel , 2010, ICCS.

[29]  Yuh-Jye Lee,et al.  RSVM: Reduced Support Vector Machines , 2001, SDM.

[30]  N. Peng,et al.  CUDA-Accelerated SVM for Celestial Object Classification , 2011 .

[31]  Peng Peng,et al.  The research of the parallel SMO algorithm for solving SVM , 2009, 2009 International Conference on Machine Learning and Cybernetics.

[32]  Jennifer G. Dy,et al.  A hierarchical method for multi-class support vector machines , 2004, ICML.

[33]  Jan Vaněk,et al.  A GPU-Architecture Optimized Hierarchical Decomposition Algorithm for Support Vector Machine Training , 2017, IEEE Transactions on Parallel and Distributed Systems.

[34]  Alessandro Sperduti,et al.  An efficient SMO-like algorithm for multiclass SVM , 2002, Proceedings of the 12th IEEE Workshop on Neural Networks for Signal Processing.

[35]  Mingsheng Hu,et al.  A parallel approach for SVM with multi-core CPU , 2010, 2010 International Conference on Computer Application and System Modeling (ICCASM 2010).

[36]  Amund Tveit,et al.  Proximal Support Vector Machine Classifier using a Heap-based Tree Topology , 2007 .

[37]  Thorsten Joachims,et al.  Making large scale SVM learning practical , 1998 .

[38]  Bin Zhao,et al.  Support Vector Machine and its Application in Handwritten Numeral Recognition , 2000, ICPR.

[39]  Steven C. H. Hoi,et al.  Cascade RSVM in Peer-to-Peer Networks , 2008, ECML/PKDD.

[40]  Yinghuan Shi,et al.  Group-Based Alternating Direction Method of Multipliers for Distributed Linear Classification , 2017, IEEE Transactions on Cybernetics.

[41]  Moreno Marzolla.  Fast training of support vector machines on the Cell processor , 2011, Neurocomputing.

[42]  Zheng Chen,et al.  P-packSVM: Parallel Primal grAdient desCent Kernel SVM , 2009, 2009 Ninth IEEE International Conference on Data Mining.

[43]  Jacek Gondzio,et al.  High-Performance Parallel Support Vector Machine Training , 2009 .

[44]  Hongle Du,et al.  A cooperative intrusion detection system based on improved parallel SVM , 2009, 2009 Joint Conferences on Pervasive Computing (JCPC).

[45]  Nilanjan Dey,et al.  A Novel Approach of Color Image Hiding using RGB Color planes and DWT , 2012, ArXiv.

[46]  Alessandro Moschitti,et al.  Fast Support Vector Machines for Structural Kernels , 2011, ECML/PKDD.

[47]  Stefan Trausan-Matu,et al.  Mapping Data Mining Algorithms on a GPU Architecture: A Study , 2011, ISMIS.

[48]  John C. Platt.  Using Analytic QP and Sparseness to Speed Training of Support Vector Machines , 1998, NIPS.

[49]  Guosheng Wang.  A Survey on Training Algorithms for Support Vector Machine Classifiers , 2008, 2008 Fourth International Conference on Networked Computing and Advanced Information Management.

[50]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[51]  John Langford,et al.  Scaling up machine learning: parallel and distributed approaches , 2011, KDD '11 Tutorials.

[52]  Himansu Sekhar Behera,et al.  A Comprehensive Survey on Support Vector Machine in Data Mining Tasks: Applications & Challenges , 2015 .

[53]  Jian-xiong Dong,et al.  A Fast Parallel Optimization for Training Support Vector Machine , 2003, MLDM.

[54]  Kunle Olukotun,et al.  Map-Reduce for Machine Learning on Multicore , 2006, NIPS.

[55]  Jing He,et al.  A survey of GPU accelerated SVM , 2014, ACM Southeast Regional Conference.

[56]  Jeyanthi Narasimhan,et al.  Fast and Accurate Support Vector Machines on Large Scale Systems , 2015, 2015 IEEE International Conference on Cluster Computing.

[57]  Stephen Tyree,et al.  Parallel Support Vector Machines in Practice , 2014, ArXiv.

[58]  Timo M. Deist,et al.  Infrastructure and distributed learning methodology for privacy-preserving multi-centric rapid learning health care: euroCAT , 2017, Clinical and translational radiation oncology.

[59]  Samy Bengio,et al.  A Parallel Mixture of SVMs for Very Large Scale Problems , 2001, Neural Computation.

[60]  Guo-Zheng Li,et al.  Incremental partial least squares analysis of big streaming data , 2014, Pattern Recognit..

[61]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[62]  Roberto Díaz-Morales,et al.  Improving the efficiency of IRWLS SVMs using parallel Cholesky factorization , 2016, Pattern Recognit. Lett..

[63]  Maozhen Li,et al.  A Resource Aware MapReduce Based Parallel SVM for Large Scale Image Classifications , 2016, Neural Processing Letters.

[64]  Maozhen Li,et al.  A distributed SVM for scalable image annotation , 2011, 2011 Eighth International Conference on Fuzzy Systems and Knowledge Discovery (FSKD).

[65]  Hao Wang,et al.  PSVM : Parallelizing Support Vector Machines on Distributed Computers , 2007 .

[66]  Terran Lane,et al.  Parallel Computation of RBF Kernels for Support Vector Classifiers , 2005, SDM.

[67]  E. Chang,et al.  Parallel algorithms for mining large-scale rich-media data , 2009, ACM Multimedia.

[68]  Qi Li,et al.  GPUSVM: a comprehensive CUDA based support vector machine package , 2011, Central European Journal of Computer Science.

[69]  François Poulet,et al.  A Fast Parallel SVM Algorithm for Massive Classification Tasks , 2008, MCO.

[70]  Ulrich Brunsmann,et al.  Gpu architecture for stationary multisensor pedestrian detection at smart intersections , 2011, 2011 IEEE Intelligent Vehicles Symposium (IV).

[71]  V. Kamakshi Prasad,et al.  Implementation of parallel algorithm for support vector machine applied to intrusion detection systems , 2016, 2016 International Conference on Computing, Analytics and Security Trends (CAST).

[72]  Maozhen Li,et al.  A MapReduce based parallel SVM for large scale spam filtering , 2011, 2011 Eighth International Conference on Fuzzy Systems and Knowledge Discovery (FSKD).

[73]  Bin Wu,et al.  Budgeted mini-batch parallel gradient descent for support vector machines on Spark , 2014, 2014 20th IEEE International Conference on Parallel and Distributed Systems (ICPADS).

[74]  Chih-Jen Lin,et al.  LIBLINEAR: A Library for Large Linear Classification , 2008, J. Mach. Learn. Res..

[75]  Minglu Li,et al.  A Peer-to-Peer Hypertext Categorization Using Directed Acyclic Graph Support Vector Machines , 2004, PDCAT.

[76]  Ovidiu Ivanciuc,et al.  Applications of Support Vector Machines in Chemistry , 2007 .

[77]  Jean-Daniel Fekete,et al.  Large Scale Classification with Support Vector Machine Algorithms , 2007, Sixth International Conference on Machine Learning and Applications (ICMLA 2007).

[78]  Tamir Hazan,et al.  A Parallel Decomposition Solver for SVM: Distributed dual ascend using Fenchel Duality , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[79]  Clive Maxfield.  FPGAs: Instant Access , 2008 .

[80]  Austin Carpenter,et al.  CUSVM: A CUDA IMPLEMENTATION OF SUPPORT VECTOR CLASSIFICATION AND REGRESSION , 2009 .

[81]  S. Sathiya Keerthi,et al.  Parallel sequential minimal optimization for the training of support vector machines , 2006, IEEE Trans. Neural Networks.

[82]  Jason Weston,et al.  Large-Scale Parallel SVM Implementation , 2007 .

[83]  S. Sathiya Keerthi,et al.  Improvements to the SMO algorithm for SVM regression , 2000, IEEE Trans. Neural Networks Learn. Syst..

[84]  Danny Dolev,et al.  A Gaussian Belief Propagation Solver for Large Scale Support Vector Machines , 2008, ArXiv.

[85]  Xiaola Lin,et al.  Efficient SVM Training Using Parallel Primal-Dual Interior Point Method on GPU , 2013, 2013 International Conference on Parallel and Distributed Computing, Applications and Technologies.

[86]  Geoffrey Fox,et al.  Study on Parallel SVM Based on MapReduce , 2012 .

[87]  Bernd Bischl,et al.  Support Vector Machines on Large Data Sets: Simple Parallel Approaches , 2012, GfKl.

[88]  Vwani P. Roychowdhury,et al.  Distributed Parallel Support Vector Machines in Strongly Connected Networks , 2008, IEEE Transactions on Neural Networks.

[89]  Xueqin Zhang,et al.  GPU Implementation of Parallel Support Vector Machine Algorithm with Applications to Detection Intruder , 2014, J. Comput..

[90]  Zhi-Ping Fan,et al.  Parallel multiple kernel learning: a hybrid alternating direction method of multipliers , 2013, Knowledge and Information Systems.

[91]  Johan A. K. Suykens,et al.  EnsembleSVM: a library for ensemble learning using support vector machines , 2014, J. Mach. Learn. Res..

[92]  Dominik Brugger,et al.  Parallel Support Vector Machines , 2006 .

[93]  Jason Weston,et al.  Large-scale kernel machines , 2007 .

[94]  Chih-Jen Lin,et al.  Large linear classification when data cannot fit in memory , 2010, KDD '10.

[95]  Maozhen Li,et al.  A MapReduce-based distributed SVM ensemble for scalable image classification and annotation , 2013, Comput. Math. Appl..

[96]  Jing Yang,et al.  A parallel SVM training algorithm on large-scale classification problems , 2005, 2005 International Conference on Machine Learning and Cybernetics.

[97]  John R. Williams,et al.  Parallel multiclass classification using SVMs on GPUs , 2010, GPGPU-3.

[98]  Kumar Shrivastava Naveeen,et al.  An Efficient Approach Parallel Support Vector Machine for Classification of Diabetes Dataset , 2011 .

[99]  Kang G. Shin,et al.  Efficient Distributed Linear Classification Algorithms via the Alternating Direction Method of Multipliers , 2012, AISTATS.

[100]  Håkan Sundell,et al.  Empirical Study of Time Efficiency and Accuracy of Support Vector Machines Using an Improved Version of PSVM , 2015, PDPTA 2015.

[101]  Luca Zanni,et al.  Parallel Software for Training Large Scale Support Vector Machines on Multiprocessor Systems , 2006, J. Mach. Learn. Res..

[102]  B. Ribeiro,et al.  GPUMLib : An Efficient Open-Source GPU Machine Learning Library , 2011 .

[103]  Han-Chieh Chao,et al.  Intelligent Systems and Applications - Proceedings of the International Computer Symposium (ICS) held at Taichung, Taiwan, December 12-14, 2014 , 2015, Frontiers in Artificial Intelligence and Applications.

[104]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[105]  Bruno Lang,et al.  Data Mining with Parallel Support Vector Machines for Classification , 2006, ADVIS.

[106]  Hua Li,et al.  GPU Acceleration of Interior Point Methods in Large Scale SVM Training , 2013, 2013 12th IEEE International Conference on Trust, Security and Privacy in Computing and Communications.

[107]  Luca Zanni,et al.  A parallel solver for large quadratic programs in training support vector machines , 2003, Parallel Comput..

[108]  Qi Li,et al.  Parallel multitask cross validation for Support Vector Machine using GPU , 2013, J. Parallel Distributed Comput..

[109]  Fuzhen Zhuang,et al.  A parallel incremental extreme SVM classifier , 2011, Neurocomputing.

[110]  João M. F. Xavier,et al.  D-ADMM: A Communication-Efficient Distributed Algorithm for Separable Optimization , 2012, IEEE Transactions on Signal Processing.

[111]  Shao-Yi Chien,et al.  Support Vector Machines on GPU with Sparse Matrix Format , 2010, 2010 Ninth International Conference on Machine Learning and Applications.

[112]  Noel Lopes,et al.  Multi-threaded Support Vector Machines for Pattern Recognition , 2012, ICONIP.

[113]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[114]  Miriam A. M. Capretz,et al.  Challenges for MapReduce in Big Data , 2014, 2014 IEEE World Congress on Services.

[115]  J. Platt.  Sequential Minimal Optimization : A Fast Algorithm for Training Support Vector Machines , 1998 .

[116]  Tao Li,et al.  HPSVM: Heterogeneous Parallel SVM with Factorization Based IPM Algorithm on CPU-GPU Cluster , 2016, 2016 24th Euromicro International Conference on Parallel, Distributed, and Network-Based Processing (PDP).

[117]  Ya-Xiang Yuan,et al.  A parallel decomposition algorithm for training multiclass kernel-based vector machines , 2011, Optim. Methods Softw..

[118]  Shuaiwen Song,et al.  Scaling Support Vector Machines on modern HPC platforms , 2015, J. Parallel Distributed Comput..

[119]  Ming Yang,et al.  Large-scale image classification: Fast feature extraction and SVM training , 2011, CVPR 2011.

[120]  Emilio Parrado-Hernández,et al.  Distributed support vector machines , 2006, IEEE Trans. Neural Networks.

[121]  John Shawe-Taylor,et al.  Generalization Performance of Support Vector Machines and Other Pattern Classifiers , 1999 .

[122]  Eugenius Kaszkurewicz,et al.  Parallel Implementation of Gradient-Based Neural Networks for SVM Training , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[123]  Shuaiwen Song,et al.  MIC-SVM: Designing a Highly Efficient Support Vector Machine for Advanced Modern Multi-core and Many-Core Architectures , 2014, 2014 IEEE 28th International Parallel and Distributed Processing Symposium.

[124]  Hyeran Byun,et al.  Applications of Support Vector Machines for Pattern Recognition: A Survey , 2002, SVM.

[125]  Qi Li,et al.  An intelligent system for accelerating parallel SVM classification problems on large datasets using GPU , 2010, 2010 10th International Conference on Intelligent Systems Design and Applications.

[126]  Jing Yang.  An Improved Cascade SVM Training Algorithm with Crossed Feedbacks , 2006, First International Multi-Symposiums on Computer and Computational Sciences (IMSCCS'06).

[127]  Srihari Cadambi,et al.  A Massively Parallel Digital Learning Processor , 2008, NIPS.

[128]  Vasant Honavar,et al.  Learning Support Vector Machines from Distributed Data Sources , 2005, AAAI.

[129]  A. Boni,et al.  A reconfigurable parallel architecture for SVM classification , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[130]  Yiyong Feng,et al.  Parallel SMO algorithm implementation based on OpenMP , 2014, 2014 IEEE International Conference on System Science and Engineering (ICSSE).

[131]  Jacek Gondzio,et al.  Hybrid MPI/OpenMP Parallel Linear Support Vector Machine Training , 2009, J. Mach. Learn. Res..

[132]  Sunghoon Kim,et al.  Performance of SSE and AVX Instruction Sets , 2012, ArXiv.

[133]  Antonio Artés-Rodríguez,et al.  Double Chunking for Solving SVMs for Very Large Datasets , 2004 .

[134]  Chia-Hua Ho,et al.  Recent Advances of Large-Scale Linear Classification , 2012, Proceedings of the IEEE.

[135]  Nathan Srebro,et al.  A GPU-tailored approach for training kernelized SVMs , 2011, KDD.

[136]  Igor Durdanovic,et al.  Parallel Support Vector Machines: The Cascade SVM , 2004, NIPS.

[137]  Alexander J. Smola,et al.  Linear support vector machines via dual cached loops , 2012, KDD.

[138]  Somnath Banerjee.  Boosting inductive transfer for text classification using wikipedia , 2007, ICMLA 2007.

[139]  B. Kavya,et al.  A Survey on SVM Classifiers for Intrusion Detection , 2014 .

[140]  Roberto Díaz-Morales,et al.  LIBIRWLS: A parallel IRWLS library for full and budgeted SVMs , 2017, Knowl. Based Syst..

[141]  Roberto Díaz-Morales,et al.  Parallel Semiparametric Support Vector Machines , 2011, The 2011 International Joint Conference on Neural Networks.

[142]  Ke Xu,et al.  A MapReduce based Parallel SVM for Email Classification , 2014, J. Networks.

[143]  J. Weston,et al.  Support Vector Machine Solvers , 2007 .

[144]  Lambert Schomaker,et al.  Evaluating automatically parallelized versions of the support vector machine , 2016, Concurr. Comput. Pract. Exp..

[145]  Dmitry Pechyony,et al.  Solving Large Scale Linear SVM with Distributed Block Minimization , 2011 .

[146]  Edward Y. Chang,et al.  Incremental approximate matrix factorization for speeding up support vector machines , 2006, KDD '06.

[147]  Edward Y. Chang,et al.  Parallel Approximate Matrix Factorization for Kernel Methods , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[148]  Minyi Guo,et al.  High-Performance Computing : Paradigm and Infrastructure , 2005 .

[149]  Frédéric Magoulès,et al.  Parallel Support Vector Machines on Multi-Core and Multiprocessor Systems , 2011 .

[150]  Stephen Winters-Hilt.  Distributed SVM learning and support vector reduction , 2017 .

[151]  François Poulet,et al.  Large Scale Visual Classification with Parallel, Imbalanced Bagging of Incremental LIBLINEAR SVM , 2013, DMIN 2013.

[152]  Yimin Wen,et al.  A Hierarchical and Parallel Method for Training Support Vector Machines , 2005, ISNN.

[153]  James Demmel,et al.  Runtime Data Layout Scheduling for Machine Learning Dataset , 2017, 2017 46th International Conference on Parallel Processing (ICPP).