Parallel batch pattern training algorithm for deep neural network

The development of parallel batch pattern training algorithm for deep multilayered neural network architecture and its parallelization efficiency research on many-core system are presented in this paper. The model of a deep neural network and batch pattern training algorithm are theoretically described. The algorithmic description of the parallel batch pattern training method is presented. Our results show high parallelization efficiency of the developed algorithm on many-core parallel system with 48 CPUs with the use of message passing parallelization technology.

[1]  Geoffrey E. Hinton.  Learning multiple layers of representation , 2007, Trends in Cognitive Sciences.

[2]  José Luis Bosque,et al.  Study of neural net training methods in parallel and distributed architectures , 2010, Future Gener. Comput. Syst..

[3]  George Bosilca,et al.  Improvement of parallelization efficiency of batch pattern BP training algorithm using Open MPI , 2010, International Conference on Conceptual Structures.

[4]  Simon Haykin,et al.  Neural Networks and Learning Machines , 2010 .

[5]  Uros Lotric,et al.  Parallel Implementations of Recurrent Neural Network Learning , 2009, ICANNGA.

[6]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[7]  George Bosilca,et al.  Efficient parallelization of batch pattern training algorithm on many-core and cluster architectures , 2013, 2013 IEEE 7th International Conference on Intelligent Data Acquisition and Advanced Computing Systems (IDAACS).

[8]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[9]  Volodymyr Turchenko,et al.  Parallel Batch Pattern Training Algorithm for MLP with Two Hidden Layers on Many-Core System , 2014, DCAI.

[10]  Rajat Raina,et al.  Large-scale deep unsupervised learning using graphics processors , 2009, ICML '09.

[11]  Pascal Vincent,et al.  The Difficulty of Training Deep Architectures and the Effect of Unsupervised Pre-Training , 2009, AISTATS.

[12]  Michal Cernanský.  Training Recurrent Neural Network Using Multistream Extended Kalman Filter on Multicore Processor and Cuda Enabled Graphic Processor Unit , 2009, ICANN.

[13]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[14]  Marco Zorzi,et al.  Parallelization of Deep Networks , 2012, ESANN.

[15]  Chien-Min Wang,et al.  Dynamic resource selection heuristics for a non-reserved bidding-based Grid environment , 2010, Future Gener. Comput. Syst..

[16]  Volodymyr Turchenko,et al.  Scalability of Enhanced Parallel Batch Pattern BP Training Algorithm on General-Purpose Supercomputers , 2010, DCAI.

[17]  Lucio Grandinetti,et al.  Technique of learning rate estimation for efficient training of MLP , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[18]  Lucio Grandinetti,et al.  Parallel batch pattern BP training algorithm of recurrent neural network , 2010, 2010 IEEE 14th International Conference on Intelligent Engineering Systems.

[19]  Yoshua Bengio,et al.  Exploring Strategies for Training Deep Neural Networks , 2009, J. Mach. Learn. Res..

[20]  Thomas Hofmann,et al.  Greedy Layer-Wise Training of Deep Networks , 2007 .