Relative density based support vector machine

Abstract As a support vector machine (SVM) has good generalization ability, it has been implemented in various applications. Yet in the process of resolving the mathematical model, it needs to compute the kernel matrix, the dimension of which is equal to the number of data points in the dataset, thereby costing a very high amount of memory. Some improved algorithms are proposed to extract the boundary of the dataset so that the number of data points participating in the training process decreases and the training process can be accelerated. But the prediction accuracy of most of these algorithms is so low that many support vectors are discarded. Moreover, those methods all need to perform the main computation by the kernel function in linear feature space, which increases the computational cost. In this paper, the concept “relative density” is proposed to extract the subset containing support vectors. The measure “relative density” is designed to be more meticulous so that the new method performs more precisely than existing methods. The proposed method makes use of the fact that it has good local characteristics to perform the computations in original space without having to use any kernel function. Therefore, efficiency is also improved. Furthermore, the proposed method can be used to detect noise data, by which an inseparable problem can be transformed into a separable problem so that cross validation can be avoided in various SVM algorithms. This is an advantage that none of the existing SVM methods has. Yet another advantage of this method is that it can be considered as a framework to be used in various SVM methods. This paper presents the details of the proposed accelerated algorithm, having a time complexity of O(n log n), that decreases training time significantly without decreasing prediction accuracy. The effectiveness and efficiency of the method is demonstrated through experiments on artificial and public datasets.

[1]  José R. Dorronsoro,et al.  On the Equivalence of the SMO and MDM Algorithms for SVM Training , 2008, ECML/PKDD.

[2]  Xiaoou Li,et al.  Support vector machine classification for large data sets via minimum enclosing ball clustering , 2008, Neurocomputing.

[3]  Tin Kam Ho,et al.  Building projectable classifiers of arbitrary complexity , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[4]  Johan A. K. Suykens,et al.  Least squares support vector machine classifiers: a large scale algorithm , 1999 .

[5]  Zhansong Wu,et al.  Online adaptive least squares support vector machine and its application in utility boiler combustion optimization systems , 2011 .

[6]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[7]  S. Sathiya Keerthi,et al.  Convergence of a Generalized SMO Algorithm for SVM Classifier Design , 2002, Machine Learning.

[8]  Wei Chu,et al.  An improved conjugate gradient scheme to the solution of least squares SVM , 2005, IEEE Transactions on Neural Networks.

[9]  Kok Seng Chua,et al.  Efficient computations for large least square support vector machine classifiers , 2003, Pattern Recognit. Lett..

[10]  Johan A. K. Suykens,et al.  A Comparison of Pruning Algorithms for Sparse Least Squares Support Vector Machines , 2004, ICONIP.

[11]  Zhenfeng Zhu,et al.  Inverse matrix-free incremental proximal support vector machine , 2012, Decis. Support Syst..

[12]  Asdrúbal López Chau,et al.  Convex and concave hulls for classification with support vector machine , 2013, Neurocomputing.

[13]  Johan A. K. Suykens,et al.  First and Second Order SMO Algorithms for LS-SVM Classifiers , 2011, Neural Processing Letters.

[14]  Ning Ye,et al.  Boundary detection and sample reduction for one-class Support Vector Machines , 2014, Neurocomputing.

[15]  Yuhua Li,et al.  Selecting training points for one-class support vector machines , 2011, Pattern Recognit. Lett..

[16]  Xiang-Yan Zeng,et al.  SMO-based pruning methods for sparse least squares support vector machines , 2005, IEEE Transactions on Neural Networks.

[17]  Kristin P. Bennett,et al.  Duality and Geometry in SVM Classifiers , 2000, ICML.

[18]  Qi Li,et al.  Sphere Support Vector Machines for large classification tasks , 2013, Neurocomputing.

[19]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[20]  S. Sathiya Keerthi,et al.  A fast iterative nearest point algorithm for support vector machine classifier design , 2000, IEEE Trans. Neural Networks Learn. Syst..

[21]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[22]  Huaguang Zhang,et al.  A fuzzy support vector machine algorithm for classification based on a novel PIM fuzzy clustering method , 2014, Neurocomputing.

[23]  Johan A. K. Suykens,et al.  Sparse approximation using least squares support vector machines , 2000, 2000 IEEE International Symposium on Circuits and Systems. Emerging Technologies for the 21st Century. Proceedings (IEEE Cat No.00CH36353).