Parallel and Distributed Structured SVM Training

Structured Support Vector Machines (structured SVMs) are a fundamental machine learning algorithm, and have solid theoretical foundation and high effectiveness in applications such as natural language parsing and computer vision. However, training structured SVMs is very time-consuming, due to the large number of constraints and inferior convergence rates, especially for large training data sets. The high cost of training structured SVMs has hindered its adoption to new applications. In this article, we aim to improve the efficiency of structured SVMs by proposing a parallel and distributed solution (namely FastSSVM) for training structured SVMs building on top of MPI and OpenMP. FastSSVM exploits a series of optimizations (e.g., optimizations on data storage and synchronization) to efficiently use the resources of the nodes in a cluster and the cores of the nodes. Moreover, FastSSVM tackles the large constraint set problem by batch processing and addresses the slow convergence challenge by adapting stop conditions based on the improvement of each iteration. We theoretically prove that our solution is guaranteed to converge to a global optimum. A comprehensive experimental study shows that FastSSVM can achieve at least four times speedup over the existing solutions, and in some cases can achieve two to three orders of magnitude speedup.

[1]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[2]  Kotagiri Ramamohanarao,et al.  MASCOT: Fast and Highly Scalable SVM Cross-Validation Using GPUs and SSDs , 2014, 2014 IEEE International Conference on Data Mining.

[3]  Alan Fern,et al.  HC-Search: A Learning Framework for Search-based Structured Prediction , 2014, J. Artif. Intell. Res..

[4]  Bingsheng He,et al.  ThunderSVM: A Fast SVM Library on GPUs and CPUs , 2018, J. Mach. Learn. Res..

[5]  Thomas Hofmann,et al.  Large Margin Methods for Structured and Interdependent Output Variables , 2005, J. Mach. Learn. Res..

[6]  Banshidhar Majhi,et al.  Monaural speech separation using GA-DNN integration scheme , 2020 .

[7]  Lauwerens Kuipers,et al.  Handbook of Mathematics , 2014 .

[8]  Thomas Hofmann,et al.  Support vector machine learning for interdependent and structured output spaces , 2004, ICML.

[9]  Mark W. Schmidt,et al.  Block-Coordinate Frank-Wolfe Optimization for Structural SVMs , 2012, ICML.

[10]  Q. Zou,et al.  Protein Folds Prediction with Hierarchical Structured SVM , 2016 .

[11]  Kotagiri Ramamohanarao,et al.  Scalable and fast SVM regression using modern hardware , 2017, World Wide Web.

[12]  Yong-Sheng Chen,et al.  Batch-normalized Maxout Network in Network , 2015, ArXiv.

[13]  Sebastian Nowozin,et al.  Structured Learning and Prediction in Computer Vision , 2011, Found. Trends Comput. Graph. Vis..

[14]  Thorsten Joachims,et al.  Training structural SVMs when exact inference is intractable , 2008, ICML '08.

[15]  Claudio Gallicchio,et al.  Enhancing deep neural networks via multiple kernel learning , 2020, Pattern Recognit..

[16]  Andrew McCallum,et al.  End-to-End Learning for Structured Prediction Energy Networks , 2017, ICML.

[17]  Mohammed Bennamoun,et al.  Deep Learning for 3D Point Clouds: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Shoba Sivapatham,et al.  Performance analysis of various training targets for improving speech quality and intelligibility , 2021, Applied Acoustics.

[19]  Ben Taskar,et al.  Structured Prediction Cascades , 2010, AISTATS.

[20]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[21]  Akira Shimazu,et al.  Semantic Parsing with Structured SVM Ensemble Classification Models , 2006, ACL.

[22]  Dan Roth,et al.  Distributed Training of Structured SVM , 2015, ArXiv.

[23]  Jing Yang,et al.  A parallel SVM training algorithm on large-scale classification problems , 2005, 2005 International Conference on Machine Learning and Cybernetics.

[24]  Vijaypal Singh Dhaka,et al.  Segmentation of handwritten words using structured support vector machine , 2019, Pattern Analysis and Applications.

[25]  Thorsten Joachims,et al.  Cutting-plane training of structural SVMs , 2009, Machine Learning.

[26]  S. Sundararajan,et al.  A Sequential Dual Method for Structural SVMs , 2011, SDM.

[27]  Lei Zhang,et al.  Object Tracking via Dual Linear Structured SVM and Explicit Feature Map , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Igor Durdanovic,et al.  Parallel Support Vector Machines: The Cascade SVM , 2004, NIPS.

[29]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[30]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[31]  Myung-Gil Jang,et al.  Fast Training of Structured SVM Using Fixed-Threshold Sequential Minimal Optimization , 2009 .

[32]  Yann LeCun,et al.  Regularization of Neural Networks using DropConnect , 2013, ICML.

[33]  Jianliang Xu,et al.  ShapeNet: A Shapelet-Neural Network Approach for Multivariate Time Series Classification , 2021, AAAI.

[34]  Bingsheng He,et al.  Efficient Multi-Class Probabilistic SVMs on GPUs , 2019, 2019 IEEE 35th International Conference on Data Engineering (ICDE).

[35]  Emilio Parrado-Hernández,et al.  Distributed support vector machines , 2006, IEEE Trans. Neural Networks.

[36]  Steve Branson,et al.  Efficient Large-Scale Structured Learning , 2013, CVPR.

[37]  Anthony Widjaja,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2003, IEEE Transactions on Neural Networks.

[38]  Dong-Hong Ji,et al.  Disorder recognition in clinical texts using multi-label structured SVM , 2017, BMC Bioinformatics.

[39]  Dan Roth,et al.  Multi-core Structural SVM Training , 2013, ECML/PKDD.

[40]  Antonio J. Plaza,et al.  Image Segmentation Using Deep Learning: A Survey , 2021, IEEE transactions on pattern analysis and machine intelligence.

[41]  Ferhat Özgür Çatak,et al.  A MapReduce based distributed SVM algorithm for binary classification , 2013, ArXiv.

[42]  Vibhav Vineet,et al.  Struck: Structured Output Tracking with Kernels , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Dinggang Shen,et al.  Early Diagnosis of Alzheimer's Disease by Joint Feature Selection and Classification on Temporally Structured Support Vector Machine , 2016, MICCAI.

[44]  Sergios Theodoridis,et al.  A geometric approach to Support Vector Machine (SVM) classification , 2006, IEEE Transactions on Neural Networks.

[45]  J. Platt.  Sequential Minimal Optimization : A Fast Algorithm for Training Support Vector Machines , 1998 .

[46]  Kurt Keutzer,et al.  Fast support vector machine training and classification on graphics processors , 2008, ICML '08.

[47]  John Langford,et al.  Learning to Search Better than Your Teacher , 2015, ICML.