Fine Tuning Deep Boltzmann Machines Through Meta-Heuristic Approaches

The Deep learning framework has been widely used in different applications from medicine to engineering. However, there is a lack of works that manage to deal with the issue of hyperparameter fine-tuning, since machine learning techniques often require a considerable human effort in this task. In this paper, we propose to fine-tune Deep Boltzmann Machines using meta-heuristic techniques, which do not require the computation of the gradient of the fitness function, that may be insurmountable in high-dimensional optimization spaces. We demonstrate the validity of the proposed approach against Deep Belief Networks concerning binary image reconstruction.

[1]  Geoffrey E. Hinton,et al.  An Efficient Learning Procedure for Deep Boltzmann Machines , 2012, Neural Computation.

[2]  João Paulo Papa,et al.  Model selection for Discriminative Restricted Boltzmann Machines through meta-heuristic techniques , 2015, J. Comput. Sci..

[3]  Honglak Lee,et al.  Learning Structured Output Representation using Deep Conditional Generative Models , 2015, NIPS.

[4]  Tijmen Tieleman,et al.  Training restricted Boltzmann machines using approximations to the likelihood gradient , 2008, ICML '08.

[5]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[6]  Radu-Emil Precup,et al.  Embedding Gravitational Search Algorithms in Convolutional Neural Networks for OCR applications , 2012, 2012 7th IEEE International Symposium on Applied Computational Intelligence and Informatics (SACI).

[7]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[8]  Tien D. Bui,et al.  Beyond Principal Components: Deep Boltzmann Machines for face modeling , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Yu Wang,et al.  Adaptive Inertia Weight Particle Swarm Optimization , 2006, ICAISC.

[10]  M. Fesanghary,et al.  An improved harmony search algorithm for solving optimization problems , 2007, Appl. Math. Comput..

[11]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[12]  Z. Geem.  Music-Inspired Harmony Search Algorithm: Theory and Applications , 2009 .

[13]  João Paulo Papa,et al.  On the Model Selection of Bernoulli Restricted Boltzmann Machines Through Harmony Search , 2015, GECCO.

[14]  João Paulo Papa,et al.  Fine-Tuning Convolutional Neural Networks Using Harmony Search , 2015, CIARP.

[15]  Xin-She Yang,et al.  Fine-tuning deep belief networks using cuckoo search , 2016 .

[16]  J. van Leeuwen,et al.  Neural Networks: Tricks of the Trade , 2002, Lecture Notes in Computer Science.

[17]  Xin-She Yang,et al.  On the Harmony Search Using Quaternions , 2016, ANNPR.

[18]  Xin-She Yang,et al.  Learning Parameters in Deep Belief Networks Through Firefly Algorithm , 2016, ANNPR.

[19]  João Paulo Papa,et al.  Fine-tuning Deep Belief Networks using Harmony Search , 2016, Appl. Soft Comput..

[20]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.