A MIC-based acceleration model of Deep Learning

In the era of Computational Intelligence developing rapidly, Deep Learning (DL) has gradually won acceptance from the world of Artificial Intelligence (AI) and it has been widely applied to the industry. However, the training of the network requires a considerable amount of time. For instances, the training of Convolution Neural Network (CNN) and Deep Belief Network (DBN) may take one week or even longer. Therefore, a new challenge has been put forward to the world of Artificial Intelligence, which demands decrease on the training time of Deep Learning algorithm effectively. And in this paper, we proposed a Deep Learning acceleration model based on MIC, which can reduce the training time significantly by using Restricted Boltzmann Machine (RBM) and Logistic Regression (LR). First, it conducts vectorization on the program, and then accelerates it by using the model we proposed in this paper. And the paper mainly consists of the design of the parallel model, which comprising data parallelism, model parallelism, a hybrid of data and model parallelism and so on. And experiments showed that the MIC-based acceleration model can reduce the training time to 1/10 of the original.

[1]  Noel Lopes,et al.  Restricted Boltzmann Machines and Deep Belief Networks on multi-core processors , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[2]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[3]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[4]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[5]  Z. Liu.  Bayes Ananlysis for Grouped Data-Gibbs Sampling Method , 1997 .

[6]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[7]  Yi Li,et al.  Mariana: Tencent Deep Learning Platform and its Applications , 2014, Proc. VLDB Endow..

[8]  Kai Yu,et al.  Large-scale deep learning at Baidu , 2013, CIKM.

[9]  Christian Igel,et al.  Training restricted Boltzmann machines: An introduction , 2014, Pattern Recognit..

[10]  Rong Gu,et al.  Training Large Scale Deep Neural Networks on the Intel Xeon Phi Many-Core Coprocessor , 2014, 2014 IEEE International Parallel & Distributed Processing Symposium Workshops.

[11]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[12]  Noel Lopes,et al.  Towards adaptive learning with improved convergence of deep belief networks on graphics processing units , 2014, Pattern Recognit..

[13]  Tao Wang,et al.  Deep learning with COTS HPC systems , 2013, ICML.

[14]  Umesh Panchaksharaiah.  How to Address Big Data , 2009 .

[15]  Jia Lei,et al.  Deep Learning: Yesterday, Today, and Tomorrow , 2013 .

[16]  Nicolas Le Roux,et al.  Representational Power of Restricted Boltzmann Machines and Deep Belief Networks , 2008, Neural Computation.