Fast pruning algorithm for multi-output LS-SVM and its application in chemical pattern classification

Classification is an important problem in chemometrics field, and least squares support vector machine (LS-SVM), which is a simplified version of standard support vector machine (SVM), is a powerful tool for classification problems. However LS-SVM loses the sparseness of ordinary SVM, which would decrease its classification speed and limit its application to chemical data mining. To keep sparseness for LS-SVM, several pruning algorithms were proposed. But few of the existent pruning algorithms can be directly applied to an LS-SVM with multiple outputs. In this paper, a new fast pruning algorithm is proposed for a multi-output LS-SVM. The support vector (SV) selection is done by linear correlation analysis based on the principal component analysis (PCA) or the classification correlative component analysis (CCCA) of kernel matrix, then the information contained in non-SV data points is transferred to SV data points, so sparseness is imposed to LS-SVM. The tests on several real-life classification problems show that the proposed sparse LS-SVM not only has an equivalently good classification performance to the standard LS-SVM but also keeps a high sparseness in model structure. How the information transferring process ensures the performance stability during pruning and the difference of SV distribution between the LS-SVM pruned by Suykens' pruning algorithm and the proposed sparse LS-SVMs are also illustrated.

[1]  G. Baudat,et al.  Kernel-based methods and function approximation , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[2]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[3]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[4]  Johan A. K. Suykens,et al.  Benchmarking Least Squares Support Vector Machine Classifiers , 2004, Machine Learning.

[5]  Correlative components analysis for pattern classification , 1996 .

[6]  Gregory J. Wolff,et al.  Optimal Brain Surgeon and general network pruning , 1993, IEEE International Conference on Neural Networks.

[7]  S. Keerthi,et al.  SMO Algorithm for Least-Squares SVM Formulations , 2003, Neural Computation.

[8]  Gavin C. Cawley,et al.  Improved sparse least-squares support vector machines , 2002, Neurocomputing.

[9]  Robert P. Cogdill,et al.  Least-Squares Support Vector Machines for Chemometrics: An Introduction and Evaluation , 2004 .

[10]  Gene H. Golub,et al.  Matrix computations , 1983 .

[11]  Joos Vandewalle,et al.  Special issue on fundamental and information processing aspects of neurocomputing , 2002, Neurocomputing.

[12]  G. Horvath,et al.  A sparse least squares support vector machine classifier , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[13]  Desire L. Massart,et al.  Reference data sets for chemometrical methods testing , 1993 .

[14]  Johan A. K. Suykens,et al.  Weighted least squares support vector machines: robustness and sparse approximation , 2002, Neurocomputing.

[15]  Hu Shangxu.  FAST PRUNING STRATEGY FOR NEURAL NETWORK SIZE OPTIMIZATION AND ITS APPLICATIONS , 2001 .

[16]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[17]  J. Roger,et al.  Application of LS-SVM to non-linear phenomena in NIR spectroscopy: development of a robust and portable sensor for acidity prediction in grapes , 2004 .

[18]  D. J. Newman,et al.  UCI Repository of Machine Learning Database , 1998 .

[19]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[20]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machines , 2002 .

[21]  Theo J. A. de Vries,et al.  Pruning error minimization in least squares support vector machines , 2003, IEEE Trans. Neural Networks.

[22]  Johan A. K. Suykens,et al.  A Comparison of Pruning Algorithms for Sparse Least Squares Support Vector Machines , 2004, ICONIP.

[23]  Xiang-Yan Zeng,et al.  SMO-based pruning methods for sparse least squares support vector machines , 2005, IEEE Transactions on Neural Networks.