Bit reduction support vector machine

Support vector machines are very accurate classifiers and have been widely used in many applications. However, the training and to a lesser extent prediction time of support vector machines on very large data sets can be very long. This paper presents a fast compression method to scale up support vector machines to large data sets. A simple bit reduction method is applied to reduce the cardinality of the data by weighting representative examples. We then develop support vector machines which may be trained on weighted data. Experiments indicate that the bit reduction support vector machine produces a significant reduction in the time required for both training and prediction with minimum loss in accuracy. It is also shown to be more accurate than random sampling, when the data is not over-compressed.

[1]  A. Winsor.  Sampling techniques. , 2000, Nursing times.

[2]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[3]  Art B. Owen,et al.  Data Squashing by Empirical Likelihood , 2004, Data Mining and Knowledge Discovery.

[4]  Jiawei Han,et al.  Classifying large data sets using SVMs with hierarchical clusters , 2003, KDD '03.

[5]  Bernhard Schölkopf,et al.  Improving the Accuracy and Speed of Support Vector Machines , 1996, NIPS.

[6]  Thomas G. Dietterich.  Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms , 1998, Neural Computation.

[7]  S. Sathiya Keerthi,et al.  Improvements to Platt's SMO Algorithm for SVM Classifier Design , 2001, Neural Computation.

[8]  Gunnar Rätsch,et al.  Input space versus feature space in kernel-based methods , 1999, IEEE Trans. Neural Networks.

[9]  Theodore Johnson,et al.  Squashing flat files flatter , 1999, KDD '99.

[10]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[11]  V. Vapnik.  Estimation of Dependences Based on Empirical Data , 2006 .

[12]  Lawrence O. Hall,et al.  Fast Accurate Fuzzy Clustering through Data Reduction , 2003 .

[13]  Christian Posse,et al.  Likelihood-Based Data Squashing: A Modeling Approach to Instance Construction , 2002, Data Mining and Knowledge Discovery.

[14]  Gunnar Rätsch,et al.  Soft Margins for AdaBoost , 2001, Machine Learning.

[15]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[16]  Matthias W. Seeger,et al.  Using the Nyström Method to Speed Up Kernel Machines , 2000, NIPS.

[17]  Daniel Boley,et al.  Training Support Vector Machines Using Adaptive Clustering , 2004, SDM.

[18]  Lawrence O. Hall,et al.  Active learning to recognize multiple types of plankton , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[19]  David J. Spiegelhalter,et al.  Machine Learning, Neural and Statistical Classification , 2009 .

[20]  D. J. Newman,et al.  UCI Repository of Machine Learning Database , 1998 .

[21]  Clifford Stein,et al.  Introduction to Algorithms, 2nd edition. , 2001 .

[22]  Christopher J. C. Burges,et al.  Simplified Support Vector Decision Rules , 1996, ICML.

[23]  Jian-xiong Dong,et al.  A Fast SVM Training Algorithm , 2003, Int. J. Pattern Recognit. Artif. Intell..

[24]  Christopher J. Merz,et al.  UCI Repository of Machine Learning Databases , 1996 .

[25]  Lawrence O. Hall,et al.  Recognizing plankton images from the shadow image particle profiling evaluation recorder , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[26]  Federico Girosi,et al.  Reducing the run-time complexity of Support Vector Machines , 1999 .

[27]  Padhraic Smyth,et al.  Towards scalable support vector machines using squashing , 2000, KDD '00.

[28]  Thorsten Joachims,et al.  Making large-scale support vector machine learning practical , 1999 .

[29]  David R. Karger,et al.  Text Bundling: Statistics Based Data-Reduction , 2003, ICML.

[30]  Katya Scheinberg,et al.  Efficient SVM Training Using Low-Rank Kernel Representations , 2002, J. Mach. Learn. Res..