Nonparallel support vector regression model and its SMO-type solver

Although the twin support vector regression (TSVR) method has been widely studied and various variants are successfully developed, the structural risk minimization (SRM) principle and model's sparseness are not given sufficient consideration. In this paper, a novel nonparallel support vector regression (NPSVR) is proposed in spirit of nonparallel support vector machine (NPSVM), which outperforms existing twin support vector regression (TSVR) methods in the following terms: (1) For each primal problem, a regularized term is added by rigidly following the SRM principle so that the kernel trick can be applied directly to the dual problems for the nonlinear case without considering an extra kernel-generated surface; (2) An ε-insensitive loss function is adopted to remain inherent sparseness as the standard support vector regression (SVR); (3) The dual problems have the same formulation with that of the standard SVR, so computing inverse matrix is well avoided and a sequential minimization optimization (SMO)-type solver is exclusively designed to accelerate the training for large-scale datasets; (4) The primal problems can approximately degenerate to those of the existing TSVRs if corresponding parameters are appropriately chosen. Numerical experiments on diverse datasets have verified the effectiveness of our proposed NPSVR in sparseness, generalization ability and scalability.

[1]  Zhiquan Qi,et al.  Efficient sparse nonparallel support vector machines for classification , 2014, Neural Computing and Applications.

[2]  Qin Zhang,et al.  ν-Nonparallel support vector machine for pattern classification , 2014, Neural Computing and Applications.

[3]  Yuan-Hai Shao,et al.  Improvements on Twin Support Vector Machines , 2011, IEEE Transactions on Neural Networks.

[4]  Divya Tomar,et al.  A comparison on multi-class classification methods based on least squares twin support vector machine , 2015, Knowl. Based Syst..

[5]  Xinjun Peng,et al.  TSVR: An efficient Twin Support Vector Machine for regression , 2010, Neural Networks.

[6]  Muhammad Tanveer.  Robust and Sparse Linear Programming Twin Support Vector Machines , 2014, Cognitive Computation.

[7]  Marie Chupin,et al.  Spatial and Anatomical Regularization of SVM: A General Framework for Neuroimaging Data , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Jing Chen,et al.  Twin support vector regression with Huber loss , 2017, J. Intell. Fuzzy Syst..

[9]  Olvi L. Mangasarian,et al.  Multisurface proximal support vector machine classification via generalized eigenvalues , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Yitian Xu,et al.  A weighted twin support vector regression , 2012, Knowl. Based Syst..

[11]  Guangyao Li,et al.  Variable stiffness composite material design by using support vector regression assisted efficient global optimization method , 2017 .

[12]  Yuan-Hai Shao,et al.  A regularization for the projection twin support vector machine , 2013, Knowl. Based Syst..

[13]  Muhammad Tanveer,et al.  A regularization on Lagrangian twin support vector regression , 2015, International Journal of Machine Learning and Cybernetics.

[14]  Yingjie Tian,et al.  Large-scale linear nonparallel support vector machine solver , 2014, Neurocomputing.

[15]  Yitian Xu,et al.  K-nearest neighbor-based weighted multi-class twin support vector machine , 2016, Neurocomputing.

[16]  Yuan-Hai Shao,et al.  MLTSVM: A novel twin support vector machine to multi-label learning , 2016, Pattern Recognit..

[17]  Yong Shi,et al.  ν-Nonparallel support vector machine for pattern classification , 2014, Neural Computing and Applications.

[18]  Jun Guo,et al.  Global Convergence of SMO Algorithm for Support Vector Regression , 2008, IEEE Transactions on Neural Networks.

[19]  Ganesh Ramakrishnan,et al.  Using Sequential Unconstrained Minimization Techniques to simplify SVM solvers , 2012, Neurocomputing.

[20]  O. Kisi,et al.  Application of least square support vector machine and multivariate adaptive regression spline models in long term prediction of river water pollution , 2016 .

[21]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[22]  Yong Shi,et al.  Ramp loss nonparallel support vector machine for pattern classification , 2015, Knowl. Based Syst..

[23]  Gary William Flake,et al.  Efficient SVM Regression Training with SMO , 2002, Machine Learning.

[24]  S. Balasundaram,et al.  Training primal twin support vector regression via unconstrained convex minimization , 2015, Applied Intelligence.

[25]  Soe W. Myint,et al.  A support vector machine to identify irrigated crop types using time-series Landsat NDVI data , 2015, Int. J. Appl. Earth Obs. Geoinformation.

[26]  Jian Yang,et al.  Smooth twin support vector regression , 2010, Neural Computing and Applications.

[27]  Xianli Pan,et al.  A Novel Twin Support-Vector Machine With Pinball Loss , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[28]  Yitian Xu,et al.  K-nearest neighbor-based weighted twin support vector regression , 2014, Applied Intelligence.

[29]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[30]  Mladen Kezunovic,et al.  Islanding Detection for Inverter-Based Distributed Generation Using Support Vector Machine Method , 2014, IEEE Transactions on Smart Grid.

[31]  James Theiler,et al.  Accurate On-line Support Vector Regression , 2003, Neural Computation.

[32]  Jason Weston,et al.  Gene Selection for Cancer Classification using Support Vector Machines , 2002, Machine Learning.

[33]  Muhammad Tanveer,et al.  An efficient regularized K-nearest neighbor based weighted twin support vector regression , 2016, Knowl. Based Syst..

[34]  Yuan-Hai Shao,et al.  An ε-twin support vector machine for regression , 2012, Neural Computing and Applications.

[35]  Reshma Khemchandani,et al.  Twin Support Vector Machines for Pattern Classification , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[37]  Jayadeva,et al.  TWSVR: Twin Support Vector Machine Based Regression , 2017 .

[38]  Fan Ye,et al.  A multi-hierarchical successive optimization method for reduction of spring-back in autoclave forming , 2018 .

[39]  Jun Guo,et al.  A Novel Sequential Minimal Optimization Algorithm for Support Vector Regression , 2006, ICONIP.

[40]  S. Sathiya Keerthi,et al.  Improvements to the SMO algorithm for SVM regression , 2000, IEEE Trans. Neural Networks Learn. Syst..

[41]  Xiaoyan Li,et al.  Asymmetric ν-twin support vector regression , 2017, Neural Computing and Applications.

[42]  S. Balasundaram,et al.  Training Lagrangian twin support vector regression via unconstrained convex minimization , 2014, Knowl. Based Syst..

[43]  Yong Shi,et al.  Robust twin support vector machine for pattern classification , 2013, Pattern Recognit..