Simultaneous Tuning of Hyperparameter and Parameter for Support Vector Machines

Automatic tuning of hyperparameter and parameter is an essential ingredient and important process for learning and applying Support Vector Machines (SVM). Previous tuning methods choose hyperparameter and parameter separately in different iteration processes, and usually search exhaustively in parameter spaces. In this paper we propose and implement a new tuning algorithm that chooses hyperparameter and parameter for SVM simultaneously and search the parameter space efficiently with a deliberate initialization of a pair of starting points. First we derive an approximate but effective radius margin bound for soft margin SVM. Then we combine multiparameters of SVM into one vector, converting the two separate tuning processes into one optimization problem. Further we discuss the implementation issue about the new tuning algorithm, and that of choosing initial points for iteration. Finally we compare the new tuning algorithm with old gradient based method and cross validation on five benchmark data sets. The experimental results demonstrate that the new tuning algorithm is effective, and usually outperforms those classical tuning algorithms.