Robust statistics-based support vector machine and its variants: a survey

Support vector machines (SVMs) are versatile learning models which are used for both classification and regression. Several authors have reported successful applications of SVM in a wide range of fields. With the continuous growth and development in machine learning using SVM, it was observed that SVM also has some limitations. This paper focuses on limitation regarding its boundary, i.e., sensitivity to noise or outliers in the dataset. Researchers have proposed many variants and extensions of SVM to make it robust. This paper gives an overview of the developments in the field of robust statistics in support vector machines and its variants. This paper includes an up to date survey of the research development in the field of robustness in SVM and its extensions. It also includes a discussion part which not only discusses the pros and cons of the proposed approaches but also highlights some important future directions in it. This paper would be helpful for researchers working in the field of robust statistics as well as supervised machine learning. This study would also encourage the researchers to work further in the development of SVM and even its variants to improve them.

[1]  Jianli Xiao,et al.  SVM and KNN ensemble learning for traffic incident detection , 2019, Physica A: Statistical Mechanics and its Applications.

[2]  Wang Jeen-Shing,et al.  A Cluster Validity Measure With Outlier Detection for Support Vector Clustering , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[3]  Hyun Joon Shin,et al.  One-class support vector machines - an application in machine fault detection and classification , 2005, Comput. Ind. Eng..

[4]  Alan L. Yuille,et al.  The Concave-Convex Procedure , 2003, Neural Computation.

[5]  Yue Fei Wang,et al.  A new outlier detection method based on OPTICS , 2019, Sustainable Cities and Society.

[6]  C.-C. Chuang,et al.  Fuzzy Weighted Support Vector Regression With a Fuzzy Partition , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[7]  Daqiang Zhang,et al.  Novel clustering-based approach for Local Outlier Detection , 2016, 2016 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS).

[8]  Olivier Chapelle,et al.  Training a Support Vector Machine in the Primal , 2007, Neural Computation.

[9]  C.-fu Lin,et al.  Fuzzy Support Vector Machines with Automatic Membership Setting , 2005 .

[10]  Mário A. T. Figueiredo,et al.  Soft clustering using weighted one-class support vector machines , 2009, Pattern Recognit..

[11]  Xin Shen,et al.  Support vector machine classifier with truncated pinball loss , 2017, Pattern Recognit..

[12]  Le Thi Hoai An,et al.  The DC (Difference of Convex Functions) Programming and DCA Revisited with DC Models of Real World Nonconvex Optimization Problems , 2005, Ann. Oper. Res..

[13]  Jian Yang,et al.  A weighted one-class support vector machine , 2016, Neurocomputing.

[14]  Jin-Shiuh Taur,et al.  A Robust Fuzzy Support Vector Machine for Two-class Pattern Classification , 2006 .

[15]  Yuan-Hai Shao,et al.  Robust Lp-norm least squares support vector regression with feature selection , 2017, Appl. Math. Comput..

[16]  Jing Zhao,et al.  Twin least squares support vector regression , 2013, Neurocomputing.

[17]  Chin-Teng Lin,et al.  Fuzzy neural network design using support vector regression for function approximation with outliers , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[18]  Chen Jing,et al.  Fault detection based on a robust one class support vector machine , 2014, Neurocomputing.

[19]  Paul D. Gader,et al.  Fuzzy SVM for noisy data: A robust membership calculation method , 2009, 2009 IEEE International Conference on Fuzzy Systems.

[20]  Yan Chen,et al.  Randomizing SVM Against Adversarial Attacks Under Uncertainty , 2018, PAKDD.

[21]  Yong Zhang,et al.  Support vector classifier based on fuzzy c-means and Mahalanobis distance , 2010, Journal of Intelligent Information Systems.

[22]  Yuan-Hai Shao,et al.  Improved Generalized Eigenvalue Proximal Support Vector Machine , 2013, IEEE Signal Processing Letters.

[23]  Yong Shi,et al.  Ramp loss least squares support vector machine , 2016, J. Comput. Sci..

[24]  Sheng-De Wang,et al.  Training algorithms for fuzzy support vector machines with noisy data , 2003, 2003 IEEE XIII Workshop on Neural Networks for Signal Processing (IEEE Cat. No.03TH8718).

[25]  Alexander Binder,et al.  Deep One-Class Classification , 2018, ICML.

[26]  Yongli Wang,et al.  Revisiting transductive support vector machines with margin distribution embedding , 2018, Knowl. Based Syst..

[27]  Zne-Jung Lee,et al.  Hybrid robust support vector machines for regression with outliers , 2011, Appl. Soft Comput..

[28]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[29]  Jing Chen,et al.  Twin support vector regression with Huber loss , 2017, J. Intell. Fuzzy Syst..

[30]  Olvi L. Mangasarian,et al.  Multisurface proximal support vector machine classification via generalized eigenvalues , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Li Sun,et al.  Weighted support vector machine based on association rules , 2013, 2013 International Conference on Machine Learning and Cybernetics.

[32]  Seyed Mojtaba Hosseini Bamakan,et al.  Ramp loss K-Support Vector Classification-Regression; a robust and sparse multi-class approach to the intrusion detection problem , 2017, Knowl. Based Syst..

[33]  Rob Law,et al.  Fuzzy support vector regression machine with penalizing Gaussian noises on triangular fuzzy number space , 2010, Expert Syst. Appl..

[34]  Seyed Mojtaba Hosseini Bamakan,et al.  Ramp loss one-class support vector machine; A robust and effective approach to anomaly detection problems , 2018, Neurocomputing.

[35]  Yuan-Hai Shao,et al.  Robust L1-norm non-parallel proximal support vector machine , 2016 .

[36]  Seo Young Park,et al.  Robust penalized logistic regression with truncated loss functions , 2011, Canadian Journal of Statistics-revue Canadienne De Statistique.

[37]  Liming Yang,et al.  A Robust Regression Framework with Laplace Kernel-Induced Loss , 2017, Neural Computation.

[38]  M. Ramu,et al.  Design and Implementation of Intelligent System to Detect Malicious Facebook Posts Using Support Vector Machine (SVM) , 2018, Soft Computing and Medical Bioinformatics.

[39]  Hava T. Siegelmann,et al.  Support Vector Clustering , 2002, J. Mach. Learn. Res..

[40]  Jean-Charles Noyer,et al.  Generalized eigenvalue proximal support vector machines for outlier description , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[41]  Yongquan Zhou,et al.  A sparse method for least squares twin support vector regression , 2016, Neurocomputing.

[42]  Vasile Palade,et al.  FSVM-CIL: Fuzzy Support Vector Machines for Class Imbalance Learning , 2010, IEEE Transactions on Fuzzy Systems.

[43]  Jie Li,et al.  Training robust support vector machine with smooth Ramp loss in the primal space , 2008, Neurocomputing.

[44]  Young-Sik Choi,et al.  Least squares one-class support vector machine , 2009, Pattern Recognit. Lett..

[45]  Panos M. Pardalos,et al.  Ramp-loss nonparallel support vector regression: Robust, sparse and scalable approximation , 2018, Knowl. Based Syst..

[46]  Divya Tomar,et al.  Feature Selection based Least Square Twin Support Vector Machine for Diagnosis of Heart Disease , 2014, BSBT 2014.

[47]  Duygu Kaya,et al.  Optimization of SVM Parameters with Hybrid CS-PSO Algorithms for Parkinson's Disease in LabVIEW Environment , 2019, Parkinson's disease.

[48]  Blaine Nelson,et al.  Support Vector Machines Under Adversarial Label Noise , 2011, ACML.

[49]  Shun-Feng Su,et al.  Robust support vector regression networks for function approximation with outliers , 2002, IEEE Trans. Neural Networks.

[50]  Jalal A. Nasiri,et al.  Energy-based model of least squares twin Support Vector Machines for human action recognition , 2014, Signal Process..

[51]  Christopher Leckie,et al.  High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning , 2016, Pattern Recognit..

[52]  Liming Yang,et al.  Support vector machine with truncated pinball loss and its application in pattern recognition , 2018, Chemometrics and Intelligent Laboratory Systems.

[53]  Xiaohang Li,et al.  Sparse least squares support vector machine with L0-norm in primal space , 2015, 2015 IEEE International Conference on Information and Automation.

[54]  Xiangyang Wang,et al.  Image denoising using nonsubsampled shearlet transform and twin support vector machines , 2014, Neural Networks.

[55]  Lu You,et al.  A New Robust Least Squares Support Vector Machine for Regression with Outliers , 2011 .

[56]  Xinjun Peng,et al.  TSVR: An efficient Twin Support Vector Machine for regression , 2010, Neural Networks.

[57]  Mangui Liang,et al.  Fuzzy support vector machine based on within-class scatter for classification problems with outliers or noises , 2013, Neurocomputing.

[58]  Sanjay Kumar,et al.  Development of Human Detection System for Security and Military Applications , 2019 .

[59]  Karim Faez,et al.  Robust voice activity detection directed by noise classification , 2015, Signal Image Video Process..

[60]  Huangang Wang,et al.  Robust one-class SVM for fault detection , 2016 .

[61]  Xianli Pan,et al.  A Novel Twin Support-Vector Machine With Pinball Loss , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[62]  Yitian Xu,et al.  A weighted twin support vector regression , 2012, Knowl. Based Syst..

[63]  Yufeng Liu,et al.  Robust Truncated Hinge Loss Support Vector Machines , 2007 .

[64]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[65]  João Luís Garcia Rosa,et al.  The use of one-class classifiers for differentiating healthy from epileptic EEG segments , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[66]  Tingquan Deng,et al.  An Adaptive Weighted One-Class SVM for Robust Outlier Detection , 2016 .

[67]  Nojun Kwak,et al.  Principal Component Analysis Based on L1-Norm Maximization , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[68]  Mohamad Jeragh,et al.  Combining Auto Encoders and One Class Support Vectors Machine for Fraudulant Credit Card Transactions Detection , 2018, 2018 Second World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4).

[69]  Xueying Zhang,et al.  Robust support vector data description for outlier detection with noise or uncertain data , 2015, Knowl. Based Syst..

[70]  Madan Gopal,et al.  Application of smoothing technique on twin support vector machines , 2008, Pattern Recognit. Lett..

[71]  J. T. Spooner,et al.  Adaptive and Learning Systems for Signal Processing, Communications, and Control , 2006 .

[72]  Johan A. K. Suykens,et al.  Weighted least squares support vector machines: robustness and sparse approximation , 2002, Neurocomputing.

[73]  Lan Bai,et al.  Twin Support Vector Machine for Clustering , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[74]  Andreu Català,et al.  K-SVCR. A support vector machine for multi-class classification , 2003, Neurocomputing.

[75]  Kin Keung Lai,et al.  A new fuzzy support vector machine to evaluate credit risk , 2005, IEEE Transactions on Fuzzy Systems.

[76]  Adel Al-Jumaily,et al.  PSO-SVM hybrid system for melanoma detection from histo-pathological images , 2019 .

[77]  Rui Li,et al.  Automatic blur type classification via ensemble SVM , 2019, Signal Process. Image Commun..

[78]  Le Thi Hoai An,et al.  Sparse semi-supervised support vector machines by DC programming and DCA , 2015, Neurocomputing.

[79]  Yuan-Hai Shao,et al.  An ε-twin support vector machine for regression , 2012, Neural Computing and Applications.

[80]  Tai-Yue Wang,et al.  Fuzzy support vector machine for multi-class text categorization , 2007, Inf. Process. Manag..

[81]  Jianguo Sun,et al.  Robust support vector regression in the primal , 2008, Neural Networks.

[82]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[83]  Zheng Cao,et al.  Robust support vector machines based on the rescaled hinge loss function , 2017, Pattern Recognition.

[84]  Pei-Yi Hao,et al.  Fuzzy one-class support vector machines , 2008, Fuzzy Sets Syst..

[85]  Zechao Li,et al.  L1-Norm Distance Minimization-Based Fast Robust Twin Support Vector $k$ -Plane Clustering , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[86]  Reshma Khemchandani,et al.  Twin Support Vector Machines for Pattern Classification , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[87]  Min Liu,et al.  Robust LS-SVR based on variational Bayesian and its applications , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[88]  Vishal M. Patel,et al.  One-Class Convolutional Neural Network , 2019, IEEE Signal Processing Letters.

[89]  Xinjun Peng,et al.  TPMSVM: A novel twin parametric-margin support vector machine for pattern recognition , 2011, Pattern Recognit..

[90]  Rui Guo,et al.  An improved ν-twin support vector machine , 2013, Applied Intelligence.

[91]  Johan A. K. Suykens,et al.  Non-parallel support vector classifiers with different loss functions , 2014, Neurocomputing.

[92]  Wenjie Hu,et al.  Robust support vector machine with bullet hole image classification , 2002 .

[93]  Madan Gopal,et al.  Least squares twin support vector machines for pattern classification , 2009, Expert Syst. Appl..

[94]  Li Li,et al.  Robust Support Vector Machine Using Least Median Loss Penalty , 2011 .

[95]  Janaina Mourão Miranda,et al.  Patient classification as an outlier detection problem: An application of the One-Class Support Vector Machine , 2011, NeuroImage.

[96]  Shrish Verma,et al.  Detection and Classification of Noise Using Bark Domain Features , 2018, ICCBN 2018.

[97]  Hakan Cevikalp,et al.  Large-scale robust transductive support vector machines , 2017, Neurocomputing.

[98]  Jue Wang,et al.  A New Fuzzy Support Vector Machine Based on the Weighted Margin , 2004, Neural Processing Letters.

[99]  Xinjun Peng,et al.  Primal twin support vector regression and its sparse approximation , 2010, Neurocomputing.

[100]  Zuherman Rustam,et al.  Kernel Spherical K-Means and Support Vector Machine for Acute Sinusitis Classification , 2019, IOP Conference Series: Materials Science and Engineering.

[101]  Reshma Khemchandani,et al.  Robust least squares twin support vector machine for human activity recognition , 2016, Appl. Soft Comput..

[102]  Milan Tuba,et al.  Bare Bones Fireworks Algorithm for Feature Selection and SVM Optimization , 2019, 2019 IEEE Congress on Evolutionary Computation (CEC).

[103]  Liming Yang,et al.  A sparse extreme learning machine framework by continuous optimization algorithms and its application in pattern recognition , 2016, Eng. Appl. Artif. Intell..

[104]  Archana S. Vaidya,et al.  A Survey on Different Unsupervised Techniques to Detect Outliers , 2015 .

[105]  Xinjun Peng,et al.  A nu-twin support vector machine (nu-TSVM) classifier and its geometric algorithms , 2010, Inf. Sci..

[106]  R. R. Rajalaxmi,et al.  A Mutated Salp Swarm Algorithm for Optimization of Support Vector Machine Parameters , 2019, 2019 5th International Conference on Advanced Computing & Communication Systems (ICACCS).

[107]  Yufeng Liu,et al.  Adaptively Weighted Large Margin Classifiers , 2013, Journal of computational and graphical statistics : a joint publication of American Statistical Association, Institute of Mathematical Statistics, Interface Foundation of North America.

[108]  Yingyuan Xiao,et al.  A New Support Vector Machine Plus with Pinball Loss , 2018, J. Classif..

[109]  XuLei Yang,et al.  Weighted support vector machine for data classification , 2005 .

[110]  Sheng-De Wang,et al.  Fuzzy support vector machines , 2002, IEEE Trans. Neural Networks.

[111]  Johan A. K. Suykens,et al.  Support Vector Machine Classifier With Pinball Loss , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[112]  Xiaowei Yang,et al.  A bilateral-truncated-loss based robust support vector machine for classification problems , 2015, Soft Comput..

[113]  Ping Zhong,et al.  Robust non-convex least squares loss function for regression with outliers , 2014, Knowl. Based Syst..

[114]  Weifeng Liu,et al.  Correntropy: Properties and Applications in Non-Gaussian Signal Processing , 2007, IEEE Transactions on Signal Processing.

[115]  Rabul Hussain Laskar,et al.  Impulse noise removal using SVM classification based fuzzy filter from gray scale images , 2016, Signal Process..

[116]  Johan A. K. Suykens,et al.  Ramp loss linear programming support vector machine , 2014, J. Mach. Learn. Res..

[117]  Ping Zhong,et al.  Training twin support vector regression via linear programming , 2012, Neural Computing and Applications.

[118]  Hong-Jie Xing,et al.  Robust one-class support vector machine with rescaled hinge loss function , 2018, Pattern Recognit..

[119]  Jian Yang,et al.  An improved robust and sparse twin support vector regression via linear programming , 2014, Soft Comput..

[120]  Joydeep Ghosh,et al.  Robust one-class clustering using hybrid global and local search , 2005, ICML.

[121]  Weifeng Liu,et al.  Adaptive and Learning Systems for Signal Processing, Communication, and Control , 2010 .

[122]  Chandan Srivastava,et al.  Support Vector Data Description , 2011 .

[123]  Huangang Wang,et al.  Ramp Loss based robust one-class SVM , 2017, Pattern Recognit. Lett..

[124]  Glenn Fung,et al.  Proximal support vector machine classifiers , 2001, KDD '01.

[125]  Chuanfa Chen,et al.  Least absolute deviation-based robust support vector regression , 2017, Knowl. Based Syst..

[126]  Jason Weston,et al.  Trading convexity for scalability , 2006, ICML.

[127]  Ding Sheng-feng.  An Improved Twin Support Vector Machine , 2012 .

[128]  Xu Chen,et al.  Ramp-based twin support vector clustering , 2018, Neural Computing and Applications.

[129]  Shie Mannor,et al.  Robust Regression and Lasso , 2008, IEEE Transactions on Information Theory.

[130]  Shutao Wang,et al.  Methane Detection Based on Improved Chicken Algorithm Optimization Support Vector Machine , 2019, Applied Sciences.

[131]  Guido Smits,et al.  Robust outlier detection using SVM regression , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[132]  V. Vijayalakshmi,et al.  KFCM Algorithm for Effective Brain Stroke Detection through SVM Classifier , 2018, 2018 IEEE International Conference on System, Computation, Automation and Networking (ICSCA).

[133]  Yong Shi,et al.  COID: A cluster–outlier iterative detection approach to multi-dimensional data analysis , 2011, Knowledge and Information Systems.

[134]  Dacheng Tao,et al.  Classification with Noisy Labels by Importance Reweighting , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[135]  Xiaowei Yang,et al.  A robust least squares support vector machine for regression and classification with noise , 2014, Neurocomputing.

[136]  W. R. Buckland,et al.  Outliers in Statistical Data , 1979 .

[137]  Jian Yang,et al.  Robust and Sparse Twin Support Vector Regression via Linear Programming , 2010, 2010 Chinese Conference on Pattern Recognition (CCPR).

[138]  Le Thi Hoai An,et al.  Solving a Class of Linearly Constrained Indefinite Quadratic Problems by D.C. Algorithms , 1997, J. Glob. Optim..

[139]  Koby Crammer,et al.  Robust Support Vector Machine Training via Convex Outlier Ablation , 2006, AAAI.

[140]  Yuan-Hai Shao,et al.  Robust Nonparallel Proximal Support Vector Machine With Lp-Norm Regularization , 2018, IEEE Access.

[141]  Yong Xia,et al.  GA-SVM based feature selection and parameter optimization in hospitalization expense modeling , 2019, Appl. Soft Comput..

[142]  Guangrong Ji,et al.  Weighted least squares twin support vector machines for pattern classification , 2010, 2010 The 2nd International Conference on Computer and Automation Engineering (ICCAE).

[143]  Panos M. Pardalos,et al.  A classification method based on generalized eigenvalue problems , 2007, Optim. Methods Softw..

[144]  Xiaowei Yang,et al.  A Kernel Fuzzy c-Means Clustering-Based Fuzzy Support Vector Machine Algorithm for Classification Problems With Outliers or Noises , 2011, IEEE Transactions on Fuzzy Systems.

[145]  Zheng Wang,et al.  A Weighted Least Squares Twin Support Vector Machine , 2014, J. Inf. Sci. Eng..

[146]  Anupam Joshi,et al.  Robust Fuzzy Clustering Methods to Support Web Mining , 1998 .