Sparse LS-SVM two-steps model selection method

Least Square Support Vector Machine (LS-SVM) converts the hinge loss function of SVM into a least square loss function which simplified the original quadratic programming training method to a linear system solving problem. Sparse LS-SVM is obtained with a pruning procedure. The performance of sparse LS-SVM depends on the selection of hyper-parameters (i.e. kernel and penalty parameters). Currently, CV and LOO are the most common methods to select hyper-parameters for LS-SVM. However, CV is computationally expensive while LOO yields a high variance of validation error which may mislead the selection of hyper-parameters. Selecting both kernel and penalty parameters simultaneously needs to search in a high dimensional parameter space. In this work, we propose a new two-step hyper-parameter selection method. Distance between Two Classes (DBTC) method is adopted to select the kernel parameters based on a maximization of between-class separation of the projected samples in the feature space. However, data distribution could not be helpful in penalty parameter selection. Therefore, we propose to select the penalty parameter via a minimization of a Localized Generalization Error to enhance the generalization capability of the LS-SVM. Experimental results comparing to existing methods show the proposed two-step method yields better LS-SVMs in term of average testing accuracies.