Heuristic Algorithm for Tuning Hyperparameters in Support Vector Regression

Tuning the hyperparameters of Support Vector Regression (SVR) is an important way to improve the generalization performance of SVR. Usually, this is done by minimizing some estimates of the generalization error. Cross Validation (CV) error is an approximately unbiased estimate of the generalization error. The grid search method is usually used to tune hyperparameters based on CV error. However, it is time costing. The variation trend of CV error changing with the hyperparameters was studied for SVR with Radial Basis Function (RBF) kernel. Based on the result, a heuristic algorithm for tuning hyperparameters was proposed. In the algorithm, a ladderlike search strategy was adopted to approximate the optimum solution first. Then a local search method was used to get the more accurate solution. Experiments on benchmark datasets illustrate that the algorithm is effective and efficient.