Least squares recursive projection twin support vector machine for classification

In this paper we formulate a least squares version of the recently proposed projection twin support vector machine (PTSVM) for binary classification. This formulation leads to extremely simple and fast algorithm, called least squares projection twin support vector machine (LSPTSVM) for generating binary classifiers. Different from PTSVM, we add a regularization term, ensuring the optimization problems in our LSPTSVM are positive definite and resulting better generalization ability. Instead of usually solving two dual problems, we solve two modified primal problems by solving two systems of linear equations whereas PTSVM need to solve two quadratic programming problems along with two systems of linear equations. Our experiments on publicly available datasets indicate that our LSPTSVM has comparable classification accuracy to that of PTSVM but with remarkably less computational time.

[1]  David R. Musicant,et al.  Lagrangian Support Vector Machines , 2001, J. Mach. Learn. Res..

[2]  Chih-Jen Lin,et al.  LIBLINEAR: A Library for Large Linear Classification , 2008, J. Mach. Learn. Res..

[3]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[4]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[5]  Jue Wang,et al.  Recursive Support Vector Machines for Dimensionality Reduction , 2008, IEEE Transactions on Neural Networks.

[6]  Ioannis Pitas,et al.  Novel Multiclass Classifiers Based on the Minimization of the Within-Class Variance , 2009, IEEE Transactions on Neural Networks.

[7]  Nai-Yang Deng,et al.  Support Vector Machines: Optimization Based Theory, Algorithms, and Extensions , 2012 .

[8]  Alessandro Verri,et al.  Pattern recognition with support vector machines : First International Workshop, SVM 2002, Niagara Falls, Canada, August 10, 2002 : proceedings , 2002 .

[9]  Ning Ye,et al.  Multi-weight vector projection support vector machines , 2010, Pattern Recognit. Lett..

[10]  Madan Gopal,et al.  Least squares twin support vector machines for pattern classification , 2009, Expert Syst. Appl..

[11]  Yuan-Hai Shao,et al.  Improvements on Twin Support Vector Machines , 2011, IEEE Transactions on Neural Networks.

[12]  B. Parlett.  The Symmetric Eigenvalue Problem , 1981 .

[13]  Li Yujian,et al.  Multiconlitron: a general piecewise linear classifier. , 2011, IEEE transactions on neural networks.

[14]  Christopher J. C. Burges,et al.  A Tutorial on Support Vector Machines for Pattern Recognition , 1998, Data Mining and Knowledge Discovery.

[15]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[16]  Jing-Yu Yang,et al.  Face recognition based on the uncorrelated discriminant transformation , 2001, Pattern Recognit..

[17]  Johan A. K. Suykens,et al.  Least squares support vector machine classifiers: a large scale algorithm , 1999 .

[18]  Olvi L. Mangasarian,et al.  Multisurface proximal support vector machine classification via generalized eigenvalues , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Haofeng Zhang,et al.  Distance difference and linear programming nonparallel plane classifier , 2011, Expert Syst. Appl..

[20]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[21]  Tong Heng Lee,et al.  Face recognition using recursive Fisher linear discriminant , 2006, IEEE Transactions on Image Processing.

[22]  Bernhard Schölkopf,et al.  Kernel Methods in Computational Biology , 2005 .

[23]  Jian Yang,et al.  Why can LDA be performed in PCA transformed space? , 2003, Pattern Recognit..

[24]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[25]  Reshma Khemchandani,et al.  Twin Support Vector Machines for Pattern Classification , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Jian Yang,et al.  Quotient vs. difference: Comparison between the two discriminant criteria , 2010, Neurocomputing.

[27]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[28]  Korris Fu-Lai Chung,et al.  On minimum class locality preserving variance support vector machine , 2010, Pattern Recognit..

[29]  Jian Yang,et al.  Recursive projection twin support vector machine via within-class variance minimization , 2011, Pattern Recognit..

[30]  Theodore B. Trafalis,et al.  Support vector machine for regression and applications to financial forecasting , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[31]  David G. Stork,et al.  Pattern Classification , 1973 .

[32]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[33]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[34]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[35]  Gene H. Golub,et al.  Matrix computations , 1983 .

[36]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[37]  Tao Jiang,et al.  Efficient and robust feature extraction by maximum margin criterion , 2003, IEEE Transactions on Neural Networks.

[38]  Glenn Fung,et al.  Multicategory Proximal Support Vector Machine Classifiers , 2005, Machine Learning.

[39]  Jian Yang,et al.  What's wrong with Fisher criterion? , 2002, Pattern Recognit..