Leave-One-Out Cross-Validation Based Model Selection Criteria for Weighted LS-SVMs

While the model parameters of many kernel learning methods are given by the solution of a convex optimisation problem, the selection of good values for the kernel and regularisation parameters, i.e. model selection, is much less straight-forward. This paper describes a simple and efficient approach to model selection for weighted least-squares support vector machines, and compares a variety of model selection criteria based on leave-one-out cross-validation. An external cross-validation procedure is used for performance estimation, with model selection performed independently in each fold to avoid selection bias. The best entry based on these methods was ranked in joint first place in the WCCI-2006 performance prediction challenge, demonstrating the effectiveness of this approach.

[1]  R. Shah,et al.  Least Squares Support Vector Machines , 2022 .

[2]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[3]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[4]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[5]  David M. Allen,et al.  The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction , 1974 .

[6]  Terry G. Seaks,et al.  SYMINV: An Algorithm for the Inversion of a Positive Definite Matrix by the Cholesky Decomposition , 1972 .

[7]  A. N. Tikhonov,et al.  Solutions of ill-posed problems , 1977 .

[8]  Gavin C. Cawley,et al.  Manipulation of prior probabilities in support vector classification , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[9]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[10]  J. W. Akitt.  Function Minimisation Using the Nelder and Mead Simplex Method with Limited Arithmetic Precision: The Self Regenerative Simplex , 1977, Comput. J..

[11]  Gavin C. Cawley,et al.  Fast exact leave-one-out cross-validation of sparse least-squares support vector machines , 2004, Neural Networks.

[12]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[13]  Licheng Jiao,et al.  Multiple Parameter Selection for LS-SVM Using Smooth Leave-One-Out Error , 2005, ISNN.

[14]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..