Adaptation of RBM Learning for Intel MIC Architecture

In the paper, the parallel realization of the Boltzmann Restricted Machine (RBM) is proposed. The implementation intends to use multicore architectures of modern CPUs and Intel Xeon Phi coprocessor. The learning procedure is based on the matrix description of RBM, where the learning samples are grouped into packages, and represented as matrices. The influence of the package size on convergence of learning, as well as on performance of computation, are studied for various number of threads, using conventional CPU and Intel Phi architecures. Our research confirms a potential usefulness of MIC parallel architecture for implementation of RBM and similar algorithms.

[1]  Alex Krizhevsky,et al.  One weird trick for parallelizing convolutional neural networks , 2014, ArXiv.

[2]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[3]  Henk Sips,et al.  Parallel and Distributed Systems Report Series Benchmarking Intel Xeon Phi to Guide Kernel Design Information about Parallel and Distributed Systems Report Series: Benchmarking Intel Xeon Phi to Guide Kernel Designwp , 2022 .

[4]  Pawel Gepner,et al.  Adaptation of MPDATA Heterogeneous Stencil Computation to Intel Xeon Phi Coprocessor , 2015, Sci. Program..

[5]  Jaroslaw Bilski,et al.  Parallel Architectures for Learning the RTRN and Elman Dynamic Neural Networks , 2015, IEEE Transactions on Parallel and Distributed Systems.

[6]  Janusz T. Starczewski,et al.  A New Three-Dimensional Facial Landmarks in Recognition , 2014, ICAISC.

[7]  Fei-Fei Li,et al.  Large-Scale Video Classification with Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[9]  Magdalena Laskowska,et al.  Mesoporous silica SBA-15 functionalized by nickel–phosphonic units: Raman and magnetic analysis , 2014 .

[10]  R. Scherer.  Neuro-fuzzy relational systems for nonlinear approximation and prediction , 2009 .

[11]  Krzysztof Patan,et al.  Optimal training strategies for locally recurrent neural networks , 2011 .

[12]  Krystian Lapa,et al.  A new method for designing neuro-fuzzy systems for nonlinear modelling with interpretability aspects , 2014, Neurocomputing.

[13]  Adam Krzyzak,et al.  Application of Support Vector Machines, Convolutional Neural Networks and Deep Belief Networks to Recognition of Partially Occluded Objects , 2014, ICAISC.

[14]  Tatiana Tambouratzis,et al.  Pulse Shape Discrimination of Neutrons and Gamma Rays Using Kohonen Artificial Neural Networks , 2013, J. Artif. Intell. Soft Comput. Res..

[15]  Ümit V. Çatalyürek,et al.  Performance Evaluation of Sparse Matrix Multiplication Kernels on Intel Xeon Phi , 2013, PPAM.

[16]  Jaroslaw Bilski,et al.  Parallel Realisation of the Recurrent RTRN Neural Network Learning , 2008, ICAISC.

[17]  Geoffrey E. Hinton.  A Practical Guide to Training Restricted Boltzmann Machines , 2012, Neural Networks: Tricks of the Trade.

[18]  Janusz T. Starczewski,et al.  Meshes vs. Depth Maps in Face Recognition Systems , 2012, ICAISC.

[19]  Jaroslaw Bilski,et al.  Parallel Approach to Learning of the Recurrent Jordan Neural Network , 2013, ICAISC.

[20]  Tomasz Galkowski,et al.  Nonparametric Function Fitting in the Presence of Nonstationary Noise , 2014, ICAISC.

[21]  Joseph Lin Chu,et al.  The Recognition Of Partially Occluded Objects with Support Vector Machines, Convolutional Neural Networks and Deep Belief Networks , 2014, J. Artif. Intell. Soft Comput. Res..

[22]  Jaroslaw Bilski,et al.  Parallel Realisation of QR Algorithm for Neural Networks Learning , 2004, ICAISC.

[23]  Pawel Gepner,et al.  Using Intel Xeon Phi Coprocessor to Accelerate Computations in MPDATA Algorithm , 2013, PPAM.

[24]  L. Rutkowski,et al.  Flexible Takagi-Sugeno fuzzy systems , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[25]  Meng Joo Er,et al.  Online Speed Profile Generation for Industrial Machine Tool Based on Neuro-fuzzy Approach , 2010, ICAISC.

[26]  Robert Nowicki,et al.  On design of flexible neuro-fuzzy systems for nonlinear modelling , 2013, Int. J. Gen. Syst..

[27]  Marcin Korytkowski,et al.  From Ensemble of Fuzzy Classifiers to Single Fuzzy Rule Base Classifier , 2006, ICAISC.

[28]  Marcin Gabryel,et al.  Object Detection by Simple Fuzzy Classifiers Generated by Boosting , 2013, ICAISC.

[29]  Janusz T. Starczewski,et al.  A New Approach to Determine Three-Dimensional Facial Landmarks , 2013, ICAISC.

[30]  Tomasz Galkowski.  Kernel Estimation of Regression Functions in the Boundary Regions , 2013, ICAISC.

[31]  Rafał Scherer,et al.  A Fuzzy Relational System with Linguistic Antecedent Certainty Factors , 2003 .

[32]  Jaroslaw Bilski,et al.  Parallel Realisation of the Recurrent Multi Layer Perceptron Learning , 2012, ICAISC.

[33]  Lukasz Laskowski,et al.  Functionalization of SBA-15 mesoporous silica by Cu-phosphonate units: Probing of synthesis route , 2014 .

[34]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[35]  Nicolas Le Roux,et al.  Representational Power of Restricted Boltzmann Machines and Deep Belief Networks , 2008, Neural Computation.

[36]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[37]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[38]  L. Rutkowski,et al.  Nonparametric fitting of multivariate functions , 1986 .

[39]  Paul Smolensky,et al.  Information processing in dynamical systems: foundations of harmony theory , 1986 .

[40]  Lukasz Szustak,et al.  Parallelization of 2D MPDATA EULAG algorithm on hybrid architectures with GPU accelerators , 2014, Parallel Comput..

[41]  Alexander I. Galushkin,et al.  The Parallel Approach to the Conjugate Gradient Learning Algorithm for the Feedforward Neural Networks , 2014, ICAISC.

[42]  Amar Ramdane-Cherif,et al.  Modeling & understanding environment using semantic agents , 2011 .

[43]  Rafal Scherer,et al.  Fuzzy Number-Based Hierarchical Fuzzy System , 2004, ICAISC.

[44]  Lukasz Laskowski,et al.  Spin-glass Implementation of a Hopfield Neural Structure , 2014, ICAISC.