The Tabu_Genetic Algorithm: A Novel Method for Hyper-Parameter Optimization of Learning Algorithms

Machine learning algorithms have been widely used to deal with a variety of practical problems such as computer vision and speech processing. But the performance of machine learning algorithms is primarily affected by their hyper-parameters, as without good hyper-parameter values the performance of these algorithms will be very poor. Unfortunately, for complex machine learning models like deep neural networks, it is very difficult to determine their hyper-parameters. Therefore, it is of great significance to develop an efficient algorithm for hyper-parameter automatic optimization. In this paper, a novel hyper-parameter optimization methodology is presented to combine the advantages of a Genetic Algorithm and Tabu Search to achieve the efficient search for hyper-parameters of learning algorithms. This method is defined as the Tabu_Genetic Algorithm. In order to verify the performance of the proposed algorithm, two sets of contrast experiments are conducted. The Tabu_Genetic Algorithm and other four methods are simultaneously used to search for good values of hyper-parameters of deep convolutional neural networks. Experimental results show that, compared to Random Search and Bayesian optimization methods, the proposed Tabu_Genetic Algorithm finds a better model in less time. Whether in a low-dimensional or high-dimensional space, the Tabu_Genetic Algorithm has better search capabilities as an effective method for finding the hyper-parameters of learning algorithms. The presented method in this paper provides a new solution for solving the hyper-parameters optimization problem of complex machine learning models, which will provide machine learning algorithms with better performance when solving practical problems.

[1]  Jeevan Kanesan,et al.  Hyper‐parameters optimisation of deep CNN architecture for vehicle logo recognition , 2018, IET Intelligent Transport Systems.

[2]  Melanie Mitchell,et al.  Genetic algorithms: An overview , 1995, Complex..

[3]  Ting Liu,et al.  Recent advances in convolutional neural networks , 2015, Pattern Recognit..

[4]  Dan Boneh,et al.  On genetic algorithms , 1995, COLT '95.

[5]  Fred W. Glover,et al.  Tabu Search - Part I , 1989, INFORMS J. Comput..

[6]  Yoshua Bengio,et al.  Algorithms for Hyper-Parameter Optimization , 2011, NIPS.

[7]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[8]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[9]  Patrick Siarry,et al.  A survey on optimization metaheuristics , 2013, Inf. Sci..

[10]  Yoshua Bengio,et al.  An empirical evaluation of deep architectures on problems with many factors of variation , 2007, ICML '07.

[11]  Guan Huang,et al.  traffic flow prediction model based on deep belief network and genetic algorithm , 2018 .

[12]  David D. Cox,et al.  Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures , 2013, ICML.

[13]  Sachin S. Talathi,et al.  Hyper-parameter optimization of deep convolutional networks for object recognition , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[14]  Achille Fokoue,et al.  An effective algorithm for hyperparameter optimization of neural networks , 2017, IBM J. Res. Dev..

[15]  José Ranilla,et al.  Hyper-parameter selection in deep neural networks using parallel particle swarm optimization , 2017, GECCO.

[16]  Chiman Kwan,et al.  Comparison of Deep Learning and Conventional Demosaicing Algorithms for Mastcam Images , 2019 .

[17]  Fabrizio Maria Maggi,et al.  Genetic algorithms for hyperparameter optimization in predictive business process monitoring , 2018, Inf. Syst..

[18]  Geoffrey E. Hinton A Practical Guide to Training Restricted Boltzmann Machines , 2012, Neural Networks: Tricks of the Trade.

[19]  Jasper Snoek,et al.  Freeze-Thaw Bayesian Optimization , 2014, ArXiv.

[20]  Steven R. Young,et al.  Optimizing deep learning hyper-parameters through an evolutionary algorithm , 2015, MLHPC@SC.

[21]  Chris Yakopcic,et al.  A State-of-the-Art Survey on Deep Learning Theory and Architectures , 2019, Electronics.

[22]  Katharina Eggensperger,et al.  Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters , 2013 .

[23]  Gang Luo,et al.  Progressive sampling-based Bayesian optimization for efficient and automatic machine learning model selection , 2017, Health Inf. Sci. Syst..

[24]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[25]  Silvia Curteanu,et al.  Multi-objective optimization of a stacked neural network using an evolutionary hyper-heuristic , 2012, Appl. Soft Comput..

[26]  Frank Hutter,et al.  CMA-ES for Hyperparameter Optimization of Deep Neural Networks , 2016, ArXiv.

[27]  Nadeem Javaid,et al.  Electricity Price and Load Forecasting using Enhanced Convolutional Neural Network and Enhanced Support Vector Regression in Smart Grids , 2019, Electronics.

[28]  Fred Glover,et al.  Tabu Search - Part II , 1989, INFORMS J. Comput..

[29]  Lawrence D. Jackel,et al.  Handwritten Digit Recognition with a Back-Propagation Network , 1989, NIPS.

[30]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[31]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[32]  Chris Eliasmith,et al.  Hyperopt-Sklearn: Automatic Hyperparameter Configuration for Scikit-Learn , 2014, SciPy.

[33]  Yongsheng Ding,et al.  A Novel Method Based on Line-Segment Visualizations for Hyper-Parameter Optimization in Deep Networks , 2018, Int. J. Pattern Recognit. Artif. Intell..

[34]  Michel Gendreau,et al.  Handbook of Metaheuristics , 2010 .

[35]  Kevin Leyton-Brown,et al.  An Efficient Approach for Assessing Hyperparameter Importance , 2014, ICML.

[36]  J. Potvin,et al.  Tabu Search , 2018, Handbook of Metaheuristics.

[37]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[38]  David Ginsbourger,et al.  Fast Computation of the Multi-Points Expected Improvement with Applications in Batch Selection , 2013, LION.

[39]  James Bergstra,et al.  Implementations of Algorithms for Hyper-Parameter Optimization , 2011 .

[40]  Ryan P. Adams,et al.  Gradient-based Hyperparameter Optimization through Reversible Learning , 2015, ICML.

[41]  Andreas Krause,et al.  Parallelizing Exploration-Exploitation Tradeoffs with Gaussian Process Bandit Optimization , 2012, ICML.