Deep Reinforcement Learning with Model-Based Acceleration for Hyperparameter Optimization

Hyperparameter optimization is a key part of AutoML. In recent years, there have been successful hyperparameter optimization algorithms. However, these methods still face several challenges, such as high cost of evaluating large models or large datasets. In this paper, we introduce a new deep reinforcement learning architecture with model-based acceleration to optimize hyperparameters for any machine learning model. In this method, an agent constructed by a Long Short-Term Memory Network aims at maximizing the expected accuracy of a machine learning model on a validation set. To speed up training, we employ a model to predict the accuracy on a validation set instead of evaluating a machine learning model. To effectively train the agent and the predictive model, Real-Predictive-Real training process is proposed. Besides, to reduce the variance, we propose a bootstrap pool to guide the exploration in the search space. The experiment was carried out by optimizing hyperparameters of two widely used machine learning models: Random Forests and XGBoost. Experimental results show that the proposed method outperforms random search, Bayesian optimization, and Tree-structured Parzen Estimator in terms of accuracy, time efficiency and stability.