Sparse algorithm for robust LSSVM in primal space

Abstract As having the closed form solution, the least squares support vector machine (LSSVM) has been widely used for classification and regression problems owing to its competitive performance compared with other types of SVMs. However, the LSSVM has two drawbacks: it is sensitive to outliers and its solution lacks sparseness. The robust LSSVM (R-LSSVM) partially overcomes the first drawback via its nonconvex truncated loss function, but it is unable to address the second drawback because its current algorithms produce dense solutions that are inefficient for training large-scale problems. In this paper, we interpret the robustness of the R-LSSVM from a re-weighted viewpoint and develop a primal R-LSSVM using the representer theorem. The new model may have a sparse solution. Then, we design a convergent sparse R-LSSVM (SR-LSSVM) algorithm to achieve a sparse solution of the primal R-LSSVM after obtaining a low-rank approximation of the kernel matrix. The new algorithm not only overcomes the two drawbacks of LSSVM simultaneously, it also has lower complexity than the existing algorithms. Therefore, it is very efficient at training large-scale problems. Numerous experimental results demonstrate that the SR-LSSVM can achieve better or comparable performance to other related algorithms in less training time, especially when used to train large-scale problems.

[1]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[2]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[3]  Gert R. G. Lanckriet,et al.  A Proof of Convergence of the Concave-Convex Procedure Using Zangwill's Theory , 2012, Neural Computation.

[4]  Lei Zhang,et al.  Sparse representation or collaborative representation: Which helps face recognition? , 2011, 2011 International Conference on Computer Vision.

[5]  Fatih Murat Porikli,et al.  Classification and Boosting with Multiple Collaborative Representations , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Johan A. K. Suykens,et al.  Robustness of Kernel Based Regression: A Comparison of Iterative Weighting Schemes , 2009, ICANN.

[7]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[8]  J. Sherman,et al.  Adjustment of an Inverse Matrix Corresponding to a Change in One Element of a Given Matrix , 1950 .

[9]  Zexi Hu,et al.  Extended compressed tracking via random projection based on MSERs and online LS-SVM learning , 2016, Pattern Recognit..

[10]  Lu You,et al.  A New Robust Least Squares Support Vector Machine for Regression with Outliers , 2011 .

[11]  Bingsheng He,et al.  On the O(1/n) Convergence Rate of the Douglas-Rachford Alternating Direction Method , 2012, SIAM J. Numer. Anal..

[12]  Fatih Murat Porikli,et al.  Connecting the dots in multi-class classification: From nearest subspace to collaborative representation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Inderjit S. Dhillon,et al.  Computationally Efficient Nyström Approximation using Fast Transforms , 2016, ICML.

[14]  Johan A. K. Suykens,et al.  Robust Support Vector Machines for Classification with Nonconvex and Smooth Losses , 2016, Neural Computation.

[15]  Shuisheng Zhou,et al.  Sparse LSSVM in Primal Using Cholesky Factorization for Large-Scale Problems , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[16]  Bernhard Schölkopf,et al.  A Generalized Representer Theorem , 2001, COLT/EuroCOLT.

[17]  Lei Zhang,et al.  A Probabilistic Collaborative Representation Based Approach for Pattern Classification , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Esin Dogantekin,et al.  A new intelligent hepatitis diagnosis system: PCA-LSSVM , 2011, Expert Syst. Appl..

[19]  Hongwei Liu,et al.  Computational Optimization and Applications Manuscript No. New Smoothing Svm Algorithm with Tight Error Bound and Efficient Reduced Techniques , 2022 .

[20]  Ping Zhong,et al.  Robust non-convex least squares loss function for regression with outliers , 2014, Knowl. Based Syst..

[21]  Xiaowei Yang,et al.  A robust least squares support vector machine for regression and classification with noise , 2014, Neurocomputing.

[22]  Alan L. Yuille,et al.  The Concave-Convex Procedure , 2003, Neural Computation.

[23]  Petros Drineas,et al.  On the Nyström Method for Approximating a Gram Matrix for Improved Kernel-Based Learning , 2005, J. Mach. Learn. Res..

[24]  Shai Ben-David,et al.  Understanding Machine Learning: From Theory to Algorithms , 2014 .

[25]  Michael I. Jordan,et al.  Predictive low-rank decomposition for kernel methods , 2005, ICML.

[26]  G. Horváth,et al.  A WEIGHTED GENERALIZED LS-SVM , 2003 .

[27]  Min Li,et al.  Improved diagnostics for the incipient faults in analog circuits using LSSVM based on PSO algorithm with Mahalanobis distance , 2014, Neurocomputing.

[28]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machines , 2002 .

[29]  James T. Kwok,et al.  Clustered Nyström Method for Large Scale Manifold Learning and Dimension Reduction , 2010, IEEE Transactions on Neural Networks.

[30]  Johan A. K. Suykens,et al.  Learning solutions to partial differential equations using LS-SVM , 2015, Neurocomputing.

[31]  D. Geman,et al.  Nonlinear Image Recovery with , 1995 .

[32]  Matthias W. Seeger,et al.  Using the Nyström Method to Speed Up Kernel Machines , 2000, NIPS.

[33]  Tin Kam Ho,et al.  Building projectable classifiers of arbitrary complexity , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[34]  Fang Liu,et al.  Coupled compressed sensing inspired sparse spatial-spectral LSSVM for hyperspectral image classification , 2015, Knowl. Based Syst..

[35]  Licheng Jiao,et al.  Fast Sparse Approximation for Least Squares Support Vector Machine , 2007, IEEE Transactions on Neural Networks.

[36]  Johan A. K. Suykens,et al.  Weighted least squares support vector machines: robustness and sparse approximation , 2002, Neurocomputing.

[37]  Xudong Jiang,et al.  Sparse and Dense Hybrid Representation via Dictionary Decomposition for Face Recognition , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Le Thi Hoai An,et al.  A Difference of Convex Functions Algorithm for Switched Linear Regression , 2014, IEEE Transactions on Automatic Control.

[39]  T. P. Dinh,et al.  Convex analysis approach to d.c. programming: Theory, Algorithm and Applications , 1997 .

[40]  Mila Nikolova,et al.  Analysis of Half-Quadratic Minimization Methods for Signal and Image Recovery , 2005, SIAM J. Sci. Comput..

[41]  Thomas S. Huang,et al.  Close the loop: Joint blind image restoration and recognition with sparse representation prior , 2011, 2011 International Conference on Computer Vision.

[42]  David R. Musicant,et al.  Successive overrelaxation for support vector machines , 1999, IEEE Trans. Neural Networks.

[43]  Johan A. K. Suykens,et al.  Sparse approximation using least squares support vector machines , 2000, 2000 IEEE International Symposium on Circuits and Systems. Emerging Technologies for the 21st Century. Proceedings (IEEE Cat No.00CH36353).