Robust non-convex least squares loss function for regression with outliers

In this paper, we propose a robust scheme for least squares support vector regression (LS-SVR), termed as RLS-SVR, which employs non-convex least squares loss function to overcome the limitation of LS-SVR that it is sensitive to outliers. Non-convex loss gives a constant penalty for any large outliers. The proposed loss function can be expressed by a difference of convex functions (DC). The resultant optimization is a DC program. It can be solved by utilizing the Concave–Convex Procedure (CCCP). RLS-SVR iteratively builds the regression function by solving a set of linear equations at one time. The proposed RLS-SVR includes the classical LS-SVR as its special case. Numerical experiments on both artificial datasets and benchmark datasets confirm the promising results of the proposed algorithm.

[1]  S. García,et al.  An Extension on "Statistical Comparisons of Classifiers over Multiple Data Sets" for all Pairwise Comparisons , 2008 .

[2]  Le Thi Hoai An,et al.  A D.C. Optimization Algorithm for Solving the Trust-Region Subproblem , 1998, SIAM J. Optim..

[3]  Zhongyi Hu,et al.  A PSO and pattern search based memetic algorithm for SVMs parameters optimization , 2013, Neurocomputing.

[4]  Zhongyi Hu,et al.  PSO-MISMO Modeling Strategy for MultiStep-Ahead Time Series Prediction , 2014, IEEE Transactions on Cybernetics.

[5]  Ping Zhong,et al.  Training robust support vector regression with smooth non-convex loss function , 2012, Optim. Methods Softw..

[6]  PETER J. ROUSSEEUW,et al.  Computing LTS Regression for Large Data Sets , 2005, Data Mining and Knowledge Discovery.

[7]  Xiaowei Yang,et al.  Robust least squares support vector machine based on recursive outlier elimination , 2010, Soft Comput..

[8]  Olivier Chapelle,et al.  Training a Support Vector Machine in the Primal , 2007, Neural Computation.

[9]  Xiaowei Yang,et al.  A heuristic weight-setting strategy and iteratively updating algorithm for weighted least-squares support vector regression , 2008, Neurocomputing.

[10]  Jason Weston,et al.  Large Scale Transductive SVMs , 2006, J. Mach. Learn. Res..

[11]  Shutao Li,et al.  Tuning SVM parameters by using a hybrid CLPSO-BFGS algorithm , 2010, Neurocomputing.

[12]  Jianping Li,et al.  A weighted Lq adaptive least squares support vector machine classifiers - Robust and sparse approximation , 2011, Expert Syst. Appl..

[13]  Xinjun Peng,et al.  TSVR: An efficient Twin Support Vector Machine for regression , 2010, Neural Networks.

[14]  Yuan-Hai Shao,et al.  Least squares twin parametric-margin support vector machine for classification , 2013, Applied Intelligence.

[15]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[16]  Satarupa Banerjee,et al.  Text classification: A least square support vector machine approach , 2007, Appl. Soft Comput..

[17]  Zhongyi Hu,et al.  Electricity Load Forecasting Using Support Vector Regression with Memetic Algorithms , 2013, TheScientificWorldJournal.

[18]  Johan A. K. Suykens,et al.  Robustness of Kernel Based Regression: A Comparison of Iterative Weighting Schemes , 2009, ICANN.

[19]  Zhongyi Hu,et al.  Beyond One-Step-Ahead Forecasting: Evaluation of Alternative Multi-Step-Ahead Forecasting Models for Crude Oil Prices , 2013, ArXiv.

[20]  Jason Weston,et al.  Trading convexity for scalability , 2006, ICML.

[21]  Johan A. K. Suykens,et al.  Weighted least squares support vector machines: robustness and sparse approximation , 2002, Neurocomputing.

[22]  Shaogang Gong,et al.  Support vector machine based multi-view face detection and recognition , 2004, Image Vis. Comput..

[23]  Zhongyi Hu,et al.  Multi-step-ahead time series prediction using multiple-output support vector regression , 2014, Neurocomputing.

[24]  Xin-She Yang,et al.  Firefly Algorithms for Multimodal Optimization , 2009, SAGA.

[25]  Le Thi Hoai An,et al.  The DC (Difference of Convex Functions) Programming and DCA Revisited with DC Models of Real World Nonconvex Optimization Problems , 2005, Ann. Oper. Res..

[26]  James Kennedy,et al.  Particle swarm optimization , 2002, Proceedings of ICNN'95 - International Conference on Neural Networks.

[27]  Muhammad Tanveer.  Robust and Sparse Linear Programming Twin Support Vector Machines , 2014, Cognitive Computation.

[28]  Ping Zhong,et al.  Training twin support vector regression via linear programming , 2012, Neural Computing and Applications.

[29]  Jianguo Sun,et al.  Robust support vector regression in the primal , 2008, Neural Networks.

[30]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[31]  Reshma Khemchandani,et al.  Twin Support Vector Machines for Pattern Classification , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[33]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[34]  Francisco Herrera,et al.  Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power , 2010, Inf. Sci..

[35]  Licheng Jiao,et al.  Recursive Finite Newton Algorithm for Support Vector Regression in the Primal , 2007, Neural Computation.

[36]  Koby Crammer,et al.  Robust Support Vector Machine Training via Convex Outlier Ablation , 2006, AAAI.

[37]  Zhongyi Hu,et al.  Multiple-output support vector regression with a firefly algorithm for interval-valued stock price index forecasting , 2014, Knowl. Based Syst..

[38]  Zhongyi Hu,et al.  Does restraining end effect matter in EMD-based modeling framework for time series prediction? Some experimental evidences , 2014, Neurocomputing.

[39]  Alan L. Yuille,et al.  The Concave-Convex Procedure , 2003, Neural Computation.

[40]  Jie Li,et al.  Training robust support vector machine with smooth Ramp loss in the primal space , 2008, Neurocomputing.

[41]  Yufeng Liu,et al.  Robust Truncated Hinge Loss Support Vector Machines , 2007 .

[42]  S. Balasundaram,et al.  On Lagrangian twin support vector regression , 2012, Neural Computing and Applications.