RPR-BP: A Deep Reinforcement Learning Method for Automatic Hyperparameter Optimization

We introduce a new deep reinforcement learning architecture - RPR-BP to optimize hyperparameter for any machine learning model on a given data set. In this method, an agent constructed by a Long Short-Term Memory Network aims at maximizing the expected accuracy of a machine learning model on a validation set. At each iteration, it selects a set of hyperparameters and uses the accuracy of the model on the validation set as the reward signal to update its internal parameters. After multiple iterations, the agent learns how to improve its decisions. However, the computation of the reward requires significant time and leads to low sample efficiency. To speed up training, we employ a neural network to predict the reward. The training process for the agent and the prediction network is divided into three phases: Real-Predictive-Real (RPR). First, the agent and the prediction network are trained by the real experience; then, the agent is trained by the reward generated from the prediction network; finally, the agent is trained again by the real experience. In this way, we can speed up training and make the agent achieve a high accuracy. Besides, to reduce the variance, we propose a Bootstrap Pool (BP) to guide the exploration in the search space. The experiment was carried out by optimizing hyperparameters of two widely used machine learning models: Random Forest and XGBoost. Experimental results show that the proposed method outperforms random search, Bayesian optimization and Tree-structured Parzen Estimator in terms of accuracy, time efficiency and stability.