MapReduce based distributed learning algorithm for Restricted Boltzmann Machine

Abstract Deep learning is recently regarded as the closest artificial intelligence model to human brain. It is about learning multiple levels of representation and abstraction that help to make sense of data such as images, sound, and text. One deep model often consists of a hierarchical architecture that has the capability to model super non-linear and stochastic problems. Restricted Boltzmann Machine (RBM) is the main constructing block of current deep networks, as most of deep architectures are built with it. Based on MapReduce framework and Hadoop distributed file system, this paper proposes a distributed algorithm for training the RBM model. Its implementation and performance are evaluated on Big Data platform-Hadoop. The main contribution of the new learning algorithm is that it solves the scalability problem that limits the development of deep learning. The intelligence growing process of human brain requires learning from Big Data. The distributed learning mechanism for RBM makes it possible to abstract sophisticated and informative features from Big Data to achieve high-level intelligence. The evaluations of the proposed learning algorithm are carried out on image inpainting and classification problems based on the BAS dataset and MNIST hand-written digits dataset.

[1]  B. Schölkopf,et al.  Modeling Human Motion Using Binary Latent Variables , 2007 .

[2]  John Langford,et al.  Scaling up machine learning: parallel and distributed approaches , 2011, KDD '11 Tutorials.

[3]  Geoffrey E. Hinton,et al.  Learning Multilevel Distributed Representations for High-Dimensional Sequences , 2007, AISTATS.

[4]  Geoffrey E. Hinton.  A Practical Guide to Training Restricted Boltzmann Machines , 2012, Neural Networks: Tricks of the Trade.

[5]  Laurenz Wiskott,et al.  Gaussian-binary restricted Boltzmann machines for modeling natural image statistics , 2014, PloS one.

[6]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[7]  Tapani Raiko,et al.  Gaussian-Bernoulli deep Boltzmann machine , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[8]  C. L. Philip Chen,et al.  An automatic setting for training restricted boltzmann machine , 2014, 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[9]  Xiaolong Wang,et al.  Active deep learning method for semi-supervised sentiment classification , 2013, Neurocomputing.

[10]  C. L. Philip Chen,et al.  Data-intensive applications, challenges, techniques and technologies: A survey on Big Data , 2014, Inf. Sci..

[11]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[12]  Geoffrey E. Hinton,et al.  An Efficient Learning Procedure for Deep Boltzmann Machines , 2012, Neural Computation.

[13]  Fuqiang Chen,et al.  Subset based deep learning for RGB-D object recognition , 2015, Neurocomputing.

[14]  Fuzhen Zhuang,et al.  Learning deep representations via extreme learning machines , 2015, Neurocomputing.

[15]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Christian Igel,et al.  Training restricted Boltzmann machines: An introduction , 2014, Pattern Recognit..

[17]  C. L. Philip Chen,et al.  Fuzzy Restricted Boltzmann Machine for the Enhancement of Deep Learning , 2015, IEEE Transactions on Fuzzy Systems.

[18]  Juha Heinanen,et al.  OF DATA INTENSIVE APPLICATIONS , 1986 .

[19]  Indranil Palit,et al.  Scalable and Parallel Boosting with MapReduce , 2012, IEEE Transactions on Knowledge and Data Engineering.

[20]  Geoffrey E. Hinton,et al.  Restricted Boltzmann machines for collaborative filtering , 2007, ICML '07.

[21]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[22]  Chin-Hui Lee,et al.  Exploiting deep neural networks for detection-based speech recognition , 2013, Neurocomputing.

[23]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[24]  David A. Forsyth,et al.  Representation Learning , 2015, Computer.

[25]  Joshua B. Tenenbaum,et al.  Learning with Hierarchical-Deep Models , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Zhang Xiong,et al.  A 3D model recognition mechanism based on deep Boltzmann machines , 2015, Neurocomputing.

[27]  Miguel Á. Carreira-Perpiñán,et al.  On Contrastive Divergence Learning , 2005, AISTATS.

[28]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.