Rank-r Updating Techniques for Fast Exact Cross-Validation

In recent years, Support Vector Machines (SVMs) have been successfully developed and have become powerful tools for pattern recognition and machine learning. Although SVMs have shown excellent classification and prediction performance in many real applications, the parameters setting is very crucial to the SVMs’ performance. The k-fold cross-validation (k-fold CV) and the leave-one-out cross-validation (LOOCV) are two popular methods to obtain the best parameters setting. However, the computational costs of them are prohibitively expensive, especially for large-scale problems. Based on the observation of Smooth Support Vector Machine (SSVM) updating from the computation point of view, we proposed two efficient updating strategies by the Sherman-MorrisonWoodbury formula to reduce the cost of finding the Hessian inverse in this work. We introduced our two updating strategies in the k-fold CV and the LOOCV. It will dramatically reduce the computational cost and still can have the exact answers. In the experiments, we demonstrated the effectiveness of SSVM with we proposed strategies on several datasets. Two different types of datasets are chosen to demonstrate the advantage of two different strategies. Our updating strategies can be applied to any learning algorithm which it solved iteratively and involved the Hessian inverse in each iteration, such as Smooth Support Vector Machine for 2Insensitive Regression (2-SSVR), Least-Square Support Vector Machine (LSSVM), Training SVM in the Primal, Second-Order Online Perceptron Algorithm and so forth.