Combining Boundary Detector and SND-SVM for Fast Learning

As a state-of-the-art multi-class supervised novelty detection method, supervised novelty detection-support vector machine (SND-SVM) is extended from one-class support vector machine (OC-SVM). It still requires to slove a more time-consuming quadratic programming (QP) whose scale is the number of training samples multiplied by the number of normal classes. In order to speed up SND-SVM learning, we propose a down sampling framework for SND-SVM. First, the learning result of SND-SVM is only decided by minor samples that have non-zero Lagrange multipliers. We point out that the potential samples with non-zero Lagrange multipliers are located in the boundary regions of each class. Second, the samples located in boundary regions can be found by a boundary detector. Therefore, any boundary detector can be incorporated into the proposed down sampling framework for SND-SVM. In this paper, we use a classical boundary detector, local outlier factor (LOF), to illustrate the effective of our down sampling framework for SND-SVM. The experiments, conducted on several benchmark datasets and synthetic datasets, show that it becomes much faster to train SND-SVM after down sampling.

[1]  Jiang-She Zhang,et al.  Reducing examples to accelerate support vector regression , 2007, Pattern Recognit. Lett..

[2]  Hans-Peter Kriegel,et al.  LoOP: local outlier probabilities , 2009, CIKM.

[3]  Shaojie Qiao,et al.  Non-Negative Matrix Factorization With Locality Constrained Adaptive Graph , 2020, IEEE Transactions on Circuits and Systems for Video Technology.

[4]  Ravi Sankar,et al.  A Survey of Intrusion Detection Systems in Wireless Sensor Networks , 2014, IEEE Communications Surveys & Tutorials.

[5]  Jian Yang,et al.  Relative density degree induced boundary detection for one-class SVM , 2016, Soft Comput..

[6]  Yugen Yi,et al.  Speedup Two-Class Supervised Outlier Detection , 2018, IEEE Access.

[7]  Robert P. W. Duin,et al.  Optimising Two-Stage Recognition Systems , 2005, Multiple Classifier Systems.

[8]  Rasmus Pagh,et al.  A near-linear time approximation algorithm for angle-based outlier detection in high-dimensional data , 2012, KDD.

[9]  Robert P. W. Duin,et al.  Support Vector Data Description , 2004, Machine Learning.

[10]  Jian Yang,et al.  Neighbors' distribution property and sample reduction for support vector machines , 2014, Appl. Soft Comput..

[11]  Joachim Denzler,et al.  Kernel Null Space Methods for Novelty Detection , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Bernhard Schölkopf,et al.  Estimating the Support of a High-Dimensional Distribution , 2001, Neural Computation.

[13]  Lorenz T. Biegler,et al.  On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming , 2006, Math. Program..

[14]  Yuhua Li,et al.  Selecting training points for one-class support vector machines , 2011, Pattern Recognit. Lett..

[15]  Jian Yang,et al.  On Selecting Effective Patterns for Fast Support Vector Regression Training , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[16]  Andrew P. Bradley,et al.  The use of the area under the ROC curve in the evaluation of machine learning algorithms , 1997, Pattern Recognit..

[17]  Salvatore J. Stolfo,et al.  A Geometric Framework for Unsupervised Anomaly Detection , 2002, Applications of Data Mining in Computer Security.

[18]  Jian Yang,et al.  Finding the samples near the decision plane for support vector learning , 2017, Inf. Sci..

[19]  Hans-Peter Kriegel,et al.  LOF: identifying density-based local outliers , 2000, SIGMOD 2000.

[20]  Hans-Peter Kriegel,et al.  Local outlier detection reconsidered: a generalized view on locality with applications to spatial, video, and network outlier detection , 2012, Data Mining and Knowledge Discovery.

[21]  Hans-Peter Kriegel,et al.  Angle-based outlier detection in high-dimensional data , 2008, KDD.

[22]  Jeff G. Schneider,et al.  Detecting anomalous records in categorical datasets , 2007, KDD '07.

[23]  Jian Yang,et al.  A weighted one-class support vector machine , 2016, Neurocomputing.

[24]  Zhihui Lai,et al.  Structured optimal graph based sparse feature extraction for semi-supervised learning , 2020, Signal Process..

[25]  S. Halgamuge,et al.  Reducing the Number of Training Samples for Fast Support Vector Machine Classification , 2004 .

[26]  Yuhua Li,et al.  Selecting Critical Patterns Based on Local Geometrical and Statistical Information , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Shamim Nemati,et al.  Semisupervised ECG Ventricular Beat Classification With Novelty Detection Based on Switching Kalman Filters , 2015, IEEE Transactions on Biomedical Engineering.

[28]  Antônio de Pádua Braga,et al.  SVM-KM: speeding SVMs learning with a priori cluster selection and k-means , 2000, Proceedings. Vol.1. Sixth Brazilian Symposium on Neural Networks.

[29]  Zhang Yi,et al.  An Efficient Representation-Based Method for Boundary Point and Outlier Detection , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[30]  David M. J. Tax,et al.  Novelty detection and multi-class classification in power distribution voltage waveforms , 2016, Expert Syst. Appl..

[31]  Jun Kong,et al.  Joint graph optimization and projection learning for dimensionality reduction , 2019, Pattern Recognit..

[32]  Lin Zhang,et al.  Discriminative low-rank preserving projection for dimensionality reduction , 2019, Appl. Soft Comput..

[33]  David A. Clifton,et al.  A review of novelty detection , 2014, Signal Process..

[34]  Ning Ye,et al.  Boundary detection and sample reduction for one-class Support Vector Machines , 2014, Neurocomputing.

[35]  Johan A. K. Suykens,et al.  Supervised Novelty Detection , 2013, 2013 IEEE Symposium on Computational Intelligence and Data Mining (CIDM).

[36]  Defeng Wang,et al.  Selecting valuable training samples for SVMs via data structure analysis , 2008, Neurocomputing.

[37]  Alexandros Nanopoulos,et al.  Reverse Nearest Neighbors in Unsupervised Distance-Based Outlier Detection , 2015, IEEE Transactions on Knowledge and Data Engineering.

[38]  Johan A. K. Suykens,et al.  Multi-Class Supervised Novelty Detection , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Robert P. W. Duin,et al.  Growing a multi-class classifier with a reject option , 2008, Pattern Recognit. Lett..

[40]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.