An evolutionary hyper-heuristic to optimise deep belief networks for image reconstruction

Abstract Deep Belief Networks (DBN) have become a powerful tools to deal with a wide range of applications. On complex tasks like image reconstruction, DBN’s performance is highly sensitive to parameter settings. Manually trying out different parameters is tedious and time consuming however often required in practice as there are not many better options. This work proposes an evolutionary hyper-heuristic framework for automatic parameter optimisation of DBN. The hyper-heuristic framework introduced here is the first of its kind in this domain. It involves a high level strategy and a pool of evolutionary operators such as crossover and mutation to generates DBN parameter settings by perturbing or modifying the current setting of a DBN. Providing a large set of operators could be beneficial to form a more effective high level strategy, but in the same time would increase the search space hence make it more difficulty to form a good strategy. To address this issue, a non-parametric statistical test is introduced to identify a subset of effective operators for different phases of the hyper-heuristic search. Three well-known image reconstruction datasets were used to evaluate the performance of the proposed framework. The results reveal that the proposed hyper-heuristic framework is very competitive when compared to the state of art methods.

[1]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[2]  Nasser R. Sabar,et al.  An Exponential Monte-Carlo algorithm for feature selection problems , 2014, Comput. Ind. Eng..

[3]  Graham Kendall,et al.  A graph coloring constructive hyper-heuristic for examination timetabling problems , 2012, Applied Intelligence.

[4]  Graham Kendall,et al.  Grammatical Evolution Hyper-Heuristic for Combinatorial Optimization Problems , 2013, IEEE Transactions on Evolutionary Computation.

[5]  Graham Kendall,et al.  A Dynamic Multiarmed Bandit-Gene Expression Programming Hyper-Heuristic for Combinatorial Optimization Problems , 2015, IEEE Transactions on Cybernetics.

[6]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[7]  Alan L. Yuille,et al.  Genetic CNN , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[8]  Nasser R. Sabar,et al.  Examination timetabling using scatter search hyper-heuristic , 2009, 2009 2nd Conference on Data Mining and Optimization.

[9]  Zhang Yi,et al.  Evolving Unsupervised Deep Neural Networks for Learning Meaningful Representations , 2017, IEEE Transactions on Evolutionary Computation.

[10]  Kunikazu Kobayashi,et al.  Time Series Forecasting Using Restricted Boltzmann Machine , 2012, ICIC.

[11]  Xin-She Yang,et al.  Learning Parameters in Deep Belief Networks Through Firefly Algorithm , 2016, ANNPR.

[12]  João Paulo Papa,et al.  Fine-tuning Deep Belief Networks using Harmony Search , 2016, Appl. Soft Comput..

[13]  Yong Yu,et al.  Efficient Architecture Search by Network Transformation , 2017, AAAI.

[14]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[15]  R Nasser.  An Exponential Monte-Carlo Local Search Algorithm for the Berth Allocation Problem , 2014 .

[16]  Z. Geem.  Music-Inspired Harmony Search Algorithm: Theory and Applications , 2009 .

[17]  Mengjie Zhang,et al.  A Particle Swarm Optimization-Based Flexible Convolutional Autoencoder for Image Classification , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[18]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Nasser R. Sabar,et al.  A constructive hyper-heuristics for rough set attribute reduction , 2010, 2010 10th International Conference on Intelligent Systems Design and Applications.

[20]  Kai Liu,et al.  Deep Boltzmann Machines Aided Design Based on Genetic Algorithms , 2014 .

[21]  Geoffrey E. Hinton.  A Practical Guide to Training Restricted Boltzmann Machines , 2012, Neural Networks: Tricks of the Trade.

[22]  Bin Wang,et al.  Evolving Deep Convolutional Neural Networks by Variable-Length Particle Swarm Optimization for Image Classification , 2018, 2018 IEEE Congress on Evolutionary Computation (CEC).

[23]  Bin Wang,et al.  A Hybrid Differential Evolution Approach to Designing Deep Convolutional Neural Networks for Image Classification , 2018, Australasian Conference on Artificial Intelligence.

[24]  Nasser R. Sabar,et al.  A math-hyper-heuristic approach for large-scale vehicle routing problems with time windows , 2015, 2015 IEEE Congress on Evolutionary Computation (CEC).

[25]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Graham Kendall,et al.  Population based Monte Carlo tree search hyper-heuristic for combinatorial optimization problems , 2015, Inf. Sci..

[27]  Tijmen Tieleman,et al.  Training restricted Boltzmann machines using approximations to the likelihood gradient , 2008, ICML '08.

[28]  Xin-She Yang,et al.  Firefly algorithm, stochastic test functions and design optimisation , 2010, Int. J. Bio Inspired Comput..

[29]  Geoffrey E. Hinton,et al.  Using fast weights to improve persistent contrastive divergence , 2009, ICML '09.

[30]  Graham Kendall,et al.  Tabu exponential Monte-Carlo with counter heuristic for examination timetabling , 2009, 2009 IEEE Symposium on Computational Intelligence in Scheduling.

[31]  Graham Kendall,et al.  Automatic Design of a Hyper-Heuristic Framework With Gene Expression Programming for Combinatorial Optimization Problems , 2015, IEEE Transactions on Evolutionary Computation.

[32]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[33]  M. Fesanghary,et al.  An improved harmony search algorithm for solving optimization problems , 2007, Appl. Math. Comput..

[34]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[35]  Graham Kendall,et al.  A Classification of Hyper-heuristic Approaches , 2010 .