Robust Lp-norm least squares support vector regression with feature selection

Lp-norm least squares support vector regression (Lp-LSSVR) is proposed for feature selection in regression.Using the absolute constraint and the Lp-norm regularization term, Lp-LSSVR performs robust against outliers.Lp-LSSVR ensures the useful features to be selected based on theoretical analysis.Lp-LSSVR only solves a series of linear equations, leading to fast training speed. In this paper, we aim a novel algorithm called robust Lp-norm least squares support vector regression (Lp-LSSVR) that is more robust than the traditional least squares support vector regression(LS-SVR). Using the absolute constraint and the Lp-norm regularization term, our Lp-LSSVR performs robust against outliers. Moreover, though the optimization problem is non-convex, the sparse solution of Lp-norm and the lower bonds for nonzero components technique ensure useful features selected by Lp-LSSVR, and it helps to find the local optimum of our Lp-LSSVR. Experimental results show that although Lp-LSSVR is more robust than least squares support vector regression (LS-SVR), and much faster than Lp-norm support vector regression (Lp-SVR) and SVR due to its equality constraint, it is slower than LS-SVR and L1-norm support vector regression (L1-SVR), it is as effective as Lp-SVR, L1-SVR, LS-SVR and SVR in both feature selection and regression.

[1]  Muhammad Tanveer.  Robust and Sparse Linear Programming Twin Support Vector Machines , 2014, Cognitive Computation.

[2]  Y. Ye,et al.  Lower Bound Theory of Nonzero Entries in Solutions of ℓ2-ℓp Minimization , 2010, SIAM J. Sci. Comput..

[3]  Robert Tibshirani,et al.  1-norm Support Vector Machines , 2003, NIPS.

[4]  Gerardus Sierksma,et al.  Linear and integer programming - theory and practice , 1999, Pure and applied mathematics.

[5]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[6]  Lan Bai,et al.  Information Technology and Quantitative Management , ITQM 2013 Exploring determinants of inflation in China based on L 1-- twin support vector regression , 2013 .

[7]  Yuan-Hai Shao,et al.  An ε-twin support vector machine for regression , 2012, Neural Computing and Applications.

[8]  J. Suykens,et al.  Generalized support vector regression: Duality and tensor-kernel representation , 2016, 1603.05876.

[9]  Ya-Fen Ye,et al.  Comparing Inflation Forecasts Using an $\varepsilon$-wavelet Twin Support Vector Regression , 2013 .

[10]  Nai-Yang Deng,et al.  Support Vector Machines: Optimization Based Theory, Algorithms, and Extensions , 2012 .

[11]  M. Yuan,et al.  Model selection and estimation in regression with grouped variables , 2006 .

[12]  Xue-feng Yan,et al.  Adaptive weighted least square support vector machine regression integrated with outlier detection and its application in QSAR , 2009 .

[13]  Gang Kou,et al.  Feature Selection for Nonlinear Kernel Support Vector Machines , 2007 .

[14]  Amaury Lendasse,et al.  Bankruptcy prediction using Extreme Learning Machine and financial expertise , 2014, Neurocomputing.

[15]  Paul S. Bradley,et al.  Feature Selection via Mathematical Programming , 1997, INFORMS J. Comput..

[16]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[17]  Ji Huang,et al.  Electromechanical equipment state forecasting based on genetic algorithm - support vector regression , 2011, Expert Syst. Appl..

[18]  Shian-Chang Huang,et al.  Chaos-based support vector regressions for exchange rate forecasting , 2010, Expert Syst. Appl..

[19]  Cheng-Lung Huang,et al.  A hybrid SOFM-SVR with a filter-based feature selection for stock market forecasting , 2009, Expert Syst. Appl..

[20]  Olvi L. Mangasarian,et al.  Absolute value equation solution via concave minimization , 2006, Optim. Lett..

[21]  Yuan-Hai Shao,et al.  Wavelet Lp-Norm Support Vector Regression with Feature Selection , 2015, J. Adv. Comput. Intell. Intell. Informatics.

[22]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[23]  Xiaowei Yang,et al.  Robust least squares support vector machine based on recursive outlier elimination , 2010, Soft Comput..

[24]  Shouyang Wang,et al.  Nonparametric bivariate copula estimation based on shape-restricted support vector regression , 2012, Knowl. Based Syst..

[25]  Johan A. K. Suykens,et al.  Kernelized Elastic Net Regularization: Generalization Bounds, and Sparse Recovery , 2016, Neural Computation.

[26]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[27]  Pei-Yi Hao,et al.  Interval regression analysis using support vector networks , 2009, Fuzzy Sets Syst..

[28]  Michel Verleysen,et al.  A graph Laplacian based approach to semi-supervised feature selection for regression problems , 2013, Neurocomputing.

[29]  Johan A. K. Suykens,et al.  Very Sparse LSSVM Reductions for Large-Scale Data , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[30]  Yuan-Hai Shao,et al.  A novel parametric-insensitive nonparallel support vector machine for regression , 2016, Neurocomputing.

[31]  Chih-Jen Lin,et al.  Asymptotic Behaviors of Support Vector Machines with Gaussian Kernel , 2003, Neural Computation.

[32]  Ying Jie Tian,et al.  Lp-norm proximal support vector machine and its applications , 2010, ICCS.

[33]  Mingcang Zhu,et al.  Housing price forecasting based on genetic algorithm and support vector machine , 2011, Expert Syst. Appl..

[34]  Dong Xu,et al.  A local information-based feature-selection algorithm for data regression , 2013, Pattern Recognit..

[35]  Yuan-Hai Shao,et al.  Mixed-norm linear support vector machine , 2012, Neural Computing and Applications.

[36]  Dewei Li,et al.  The Support Vector Regression with Adaptive Norms , 2013, ICCS.

[37]  Zhi-Ping Fan,et al.  Dynamic customer lifetime value prediction using longitudinal data: An improved multiple kernel SVR approach , 2013, Knowl. Based Syst..

[38]  Christopher J. C. Burges,et al.  A Tutorial on Support Vector Machines for Pattern Recognition , 1998, Data Mining and Knowledge Discovery.

[39]  Yuan-Hai Shao,et al.  Financial Conditions Index Construction Through Weighted Lp-Norm Support Vector Regression , 2015, J. Adv. Comput. Intell. Intell. Informatics.

[40]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[41]  Alexander J. Smola,et al.  Support Vector Regression Machines , 1996, NIPS.

[42]  Paul S. Bradley,et al.  Feature Selection via Concave Minimization and Support Vector Machines , 1998, ICML.

[43]  Johan A. K. Suykens,et al.  Least squares support vector machine classifiers: a large scale algorithm , 1999 .

[44]  J. Stock,et al.  Forecasting Output and Inflation: The Role of Asset Prices , 2001 .

[45]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[46]  P. Bühlmann,et al.  The group lasso for logistic regression , 2008 .

[47]  Olvi L. Mangasarian,et al.  Absolute value programming , 2007, Comput. Optim. Appl..

[48]  Jian-Bo Yang,et al.  Feature Selection Using Probabilistic Prediction of Support Vector Regression , 2011, IEEE Transactions on Neural Networks.

[49]  O. Mangasarian.  Minimum-support solutions of polyhedral concave programs * , 1999 .

[50]  Zhiqiang Zhang,et al.  Adaptive feature selection via a new version of support vector machine , 2012, Neural Computing and Applications.

[51]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[52]  Ling Zhuang,et al.  Prediction of silicon content in hot metal using support vector regression based on chaos particle swarm optimization , 2009, Expert Syst. Appl..

[53]  Johan A. K. Suykens,et al.  Weighted least squares support vector machines: robustness and sparse approximation , 2002, Neurocomputing.